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Preface 


Meeting  the  needs  of  the  Army’s  Future  Foree  requires  ongoing  research  investments  that  also 
support  the  rapid  transition  of  innovative  technologies  from  cutting-edge  basic  research  to 
fieldable  technological  solutions.  As  a  means  for  enabling  this  process,  the  Army  must  bring 
together  world-class  research  and  development  talent  and  focus  it  on  Army-specific  technology 
objectives  for  application  to  Army  needs. 

A  Collaborative  Technology  Alliance  (CTA)  is  a  partnership  among  Army  laboratories  and 
centers,  private  industry,  and  academia.  Active  collaboration  among  these  three  components  is  a 
key  element  of  the  CTA  concept,  with  each  bringing  distinctly  different  approaches, 
perspectives,  and  capabilities:  academic  research  laboratories  are  the  country’s  storehouse  of 
basic  science  innovation;  industrial  partners  have  unique  capabilities  and  relationships  that  are 
needed  to  leverage  existing  research  results  into  technology  transition;  and  Army  scientists  and 
engineers  bring  both  research  skills  and  a  Soldier-centric  focus  to  keep  programs  oriented  toward 
solving  complex  Army  technology  problems.  In  this  way,  CTAs  are  comprised  of 
multidisciplinary  teams  that  are  generating  the  technological  advances  needed  to  meet  the  needs 
of  the  Future  Force. 

The  U.S.  Army  Research  Laboratory  (ARL)  has  a  history  of  successful  CTA  collaborations 
dating  back  over  a  decade,  including  five  completed  CTA  programs.  The  Advanced  Decision 
Architectures  CTA,  which  designed,  tested,  and  transitioned  new  computer  science  innovations 
to  facilitate  better  Soldier  understanding  and  situational  understanding;  the  Advanced  Sensors 
CTA,  which  developed  affordable  sensors  to  provide  continuous  situational  awareness;  rapid, 
precise  discrimination  and  targeting;  and  the  ability  to  sense  the  environment  for  rapid 
navigation  and  defense;  the  Communications  and  Networks  CTA,  which  developed  technologies 
to  enable  a  fully  mobile,  fully  communicating,  agile,  situationally  aware,  and  survivable, 
lightweight  force,  through  advanced  wireless  networks  across  multiple,  heterogeneous  agents 
under  severe  bandwidth,  energy,  security,  and  operational  constraints;  the  Power  and  Energy 
CTA,  which  advanced  fundamental  sciences  and  understandings  of  efficient,  lightweight, 
compact  power  and  propulsion  technologies  for  the  dismounted  Soldier,  vehicles,  and  robotic 
platforms  of  the  future  Army;  and  finally,  the  Robotics  CTA,  which  researched  core  technologies 
to  support  Army  autonomous  vehicle  goals  and  advancing  the  state-of-the-art  in  perception 
technologies,  intelligent  controls,  and  human-machine  interfaces. 

Now,  the  Cognition  and  Neuroergonomics  (CaN)  CTA  seeks  to  continue  the  successes  of 
previous  collaborative  work  involving  Government,  industry,  and  academia.  The  CaN  CTA  has 
been  established  to  address  the  development  of  fundamental  translational  principles,  i.e., 
principles  governing  the  application  of  neuroscience-based  research  and  theory  to  complex 
operational  settings.  The  Alliance  will  be  building  upon  and  leverage  the  massive  worldwide 
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research  and  technology  development  efforts  in  the  cognitive  neurosciences.  For  example,  the 
popular  Web  site  scimagojr.com  reports  that  there  have  been  over  400,000  citable  articles 
published  in  the  neurosciences  worldwide  in  the  period  between  1996  and  2010,  with  articles 
appearing  in  more  than  300  peer-reviewed  neuroscience  journals.  In  addition,  the  National 
Industrial  Organization  reports  an  over  $140  billion  USD  annual  investment  in 
neurotechnologies,  and  the  publication  of  articles  related  to  brain-computer  interface  has 
witnessed  nearly  exponential  growth  over  the  past  decade. 

Exciting  new  developments  in  sensor  technologies,  statistical  modeling  approaches,  and 
computational  analysis  capabilities  are  bringing  new  methodological  and  paradigmatic  changes 
that  are  consistent  with  the  growing  appreciation  of  the  fundamental  need  to  study  human 
cognitive  behavior  under  real-world  conditions.  Furthering  the  development  and  application  of 
these  technologies,  methods,  and  approaches  are  foundational  to  the  vision  and  the  approach 
adopted  by  the  CaN  CTA,  which  is  articulated  in  this  document.  Realizing  this  vision  through 
the  focused  research  and  development  efforts  planned  within  the  CaN  CTA  has  the  potential  to 
achieve  ground-breaking  advances  in  Soldier-System  design  improvements  and  position  the 
Army  as  a  world- wide  leader  in  the  neuroscience  research  and  neurotechnology  development 
communities. 

Other  Government  agencies  are  invited  to  join  this  Alliance,  contribute  their  technical  expertise 
and  personnel,  and  participate  fully  in  collaborative  research  efforts  under  the  CaN  CTA.  The 
intellectual  synergies  established  through  such  collaboration,  including  the  sharing  of  equipment 
and  facilities  to  promote  efficiencies,  will  create  a  critical  mass  of  private  sector  and  Government 
scientists  and  engineers  focused  on  solving  the  military  technology  challenges  impacting  Soldier 
neurocognitive  performance,  as  well  as  supporting  and  stimulating  dual-use  applications  of  this 
research  and  technology  to  benefit  commercial  use.  To  achieve  this,  the  CaN  CTA  will 
implement  computational  modeling,  and  will  conduct  and  link  neuroscience-based  research  from 
multiple  levels  to  produce  advances  in  fundamental  science  and  technology,  demonstrate  and 
transition  technology,  and  develop  research  demonstrators  for  Warfighter  experimentation. 


Intentionally  left  blank. 


1.  Introduction 


Difficulties  experienced  as  U.S.  forces  attempt  to  identify  and  neutralize  the  threats  associated 
with  the  evolving  security  context  have  inspired  the  Army  to  reconsider,  at  a  fundamental  level, 
the  capabilities  and  readiness  of  its  personnel  and  materiel  resources  (Geren  and  Casey,  2008; 
Speakes  and  Martin,  2008).  Imperatives  to  prepare  and  transform  the  Army  to  meet  the  demands 
of  the  modern  strategic  environment  have  placed  science  and  technology  in  a  prominent  position. 

Enabling  technology  advances  are  enhancing  Soldier-system  performance  and  expanding 
operational  capabilities,  however,  these  advances  can  also  intensify  the  need  to  assess  the 
Soldier’s  ability  to  perform  tasks  under  complex,  dynamic,  and  time -pressured  operational 
conditions.  Technological  advances,  particularly  in  sensor  deployment,  information  bandwidth, 
and  automation,  coupled  with  economic  and  political  realities,  will  continue  to  place  more  and 
more  responsibility  on  fast,  distributed,  and  effectively  independent  decisions  by  solo  or  small 
groups  of  Soldiers  who  control  ever  more  potent  defensive  and  offensive  assets.  Under  such 
circumstances.  Soldier  cognitive  failures  in  comprehension  and  decision  making  based  on  an 
ever  more  complex  data  stream  may  become  a  critical  bottleneck  in  Army  defensive  and 
offensive  capabilities.  Indeed,  both  failures  to  act  and  imprudent  reactions  can  have  high  costs  in 
terms  of  mission  success,  human  suffering,  and  sociopolitical  perception. 

It  thus  becomes  increasingly  important  that  Army  systems  integrate  knowledge  of,  as  well  as 
actively  enhance,  operator  cognitive  state  and  reactions  to  events.  Towards  this  end, 
neuroscience-based  approaches  have  the  potential  to  provide  revolutionary  advances  to  foster 
practical  solutions  to  address  Army  needs  (cf..  National  Research  Council,  2009).  Recent 
progress  in  the  neurosciences  has  greatly  advanced  our  knowledge  of  how  brain  function 
underlies  behavior,  providing  our  modern  scientific  foundations  for  understanding  how  we  sense, 
perceive,  and  interact  with  the  external  world;  an  understanding  that,  if  properly  leveraged,  can 
lead  to  improved  capacity  for  integrating  human  neurocognitive  function  with  Army  system 
design  and  performance. 

The  Cognition  and  Neuroergonomics  (CaN)  Collaborative  Technology  Alliance  (CTA)  has 
recently  been  formed  by  the  U.S.  Army  Research  Laboratory  (ARE)  and  a  consortium  of  leading 
industry  and  academic  research  partners  to  address  these  challenges.  Here,  the  term 
“neuroergonomics”  is  used  as  originally  proposed  and  defined  by  Parasuraman  (2003)  as  “the 
study  of  the  brain  and  body  at  work,”  and  specified  for  military  applications  as  “operational 
neuroergonomics”  within  the  CaN  CTA  as  building  upon  neuroscience,  human  factors, 
psychology,  and  engineering  to  enhance  our  understanding  of  Soldier  brain  function  and 
behavior  in  complex  operational  settings,  assessed  outside  the  confines  of  standard  research 
laboratories. 
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However,  applying  basic  research  in  cognitive  science  to  the  understandings  of  Soldier 
neurocognitive  behavior  that  are  needed  to  enable  a  neuroscience-based  approach  to  systems 
design  is  non-trivial.  Several  scientific  and  technological  barriers  have  placed  significant 
limitations  on  the  impact  of  previous  research  efforts  (see  section  2.2  for  more  detailed 
discussions  of  each  scientific  barrier): 

B.l  The  restriction  of  experimental  designs  to  highly  controlled  and  impoverished 
stimulus/response  paradigms  and  environments 

B.2  The  lack  of  portable,  user-acceptable  (e.g.,  comfortably  wearable),  and  robust  systems  for 
routinely  monitoring  brain  and  body  dynamics 

B.3  The  failure  to  record  the  whole  of  physical,  mental,  and  physiological  behavior  that  the 
brain  controls,  and  the  physical  and  socio-cultural  effects  of  the  environment  that  impact 
brain  function,  in  sufficient  detail  and  across  a  sufficient  breadth  of  circumstances 

B.4  The  lack  of  mathematical  modeling  methods  to  find  statistical  relationships  among  the 
moment-to-moment  variations  in  environmental,  behavioral,  and  functional  brain 
dynamics 

B.5  The  lack  of  sufficient  data  archives  and  resources  to  systematically  study  relationships 
between  individualized  models  derived  for  cognitive  monitoring  and  individual 
differences  in  performance,  cognitive  ability,  and  personality,  relationships  that  could 
optimize  performance  of  cognitive  monitoring  systems  while  lowering  requirements  for 
collecting  individual  subject  training  data 

The  limitations  that  these  barriers  impose  have  resulted  in  a  lack  of  a  clear  working 
understanding  of  how  the  human  brain  functions  when  faced  with  real-world  tasks  in  real-world 
operational  settings,  as  opposed  to  how  it  works  in  highly  constrained  tasks  in  highly  controlled 
laboratory  settings.  In  turn,  this  has  meant  that  previous  systems  development  approaches  and 
methods  for  technological  advancement  cannot  adequately  account  for  the  full  spectrum  of 
capabilities  and  limitations  of  human  neurocognitive  function. 

Within  this  context,  the  scientific  vision  for  the  CaN  CTA  has  been  derived  from  an  ecological 
perspective,  as  well  as  concepts  of  embodied  or  situated  cognition,  that  considers  humans  to  be 
active  agents  embedded  in  a  complex  and  dynamic  environment.  Cognitive  processes,  and  the 
brain  function  that  underlies  them,  have  evolved  to  optimize  the  outcome  of  our  behaviors  within 
our  physical  and  socio-cultural  environments.  Acceptance  of  this  position  imposes  specific 
constraints  on  our  methodological  decisions;  in  particular,  this  understanding  indicates  that  it  is 
literally  impossible  to  understand  natural,  motivated  behavior  in  artificial  scenarios  that  divorce 
the  individual  from  their  context. 
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New  approaches  to  neuroscientific  inquiry  are  therefore  needed  to  overcome  the  scientific  and 
technological  barriers  that  have  limited  previous  research  efforts.  The  approach  taken  by  the 
CaN  CTA  aims  to  address  these  barriers  directly  through  the  following: 

•  Development  of  experimental  paradigms  that  capture  the  unfolding  nature  of  multisensory 
stimulus  streams  experienced  in  real-world  environments 

•  Development  and  employment  of  novel,  wearable  sensor  suites  for  monitoring  brain  and 
body  dynamics  during  naturalistic  behavior  as  well  as  the  software  for  integrating  brain, 
sensory  and  behavioral  monitoring  systems 

•  Acquisition  and  processing  of  high-dimensional  datasets  that  characterize  physical,  mental, 
and  physiological  behavior,  as  well  as  its  environmental  context,  in  sufficient  detail  across 
a  sufficient  breadth  of  circumstances 

•  Discovery  of  models  and  novel  methods  for  the  identification  and  interpretation  of 
statistical  relationships  among  high-dimensional  data  sets  characterizing  the  dynamics  of 
environment,  behavior,  and  brain  function  during  complex  task  performance 

•  The  acquisition  and  analysis  of  data  from  a  large  number  of  individuals  allowing 
characterization  of  inter-  and  intra-individual  variation  in  order  to  systematically  study  the 
relationships  between  individualized  models  derived  for  cognitive  monitoring  and 
individual  differences  in  performance,  cognitive  ability,  and  personality 

Success  within  this  approach,  in  turn,  is  envisioned  to  lead  to  the  establishment  of  principles  on 
which  to  base  advances  in  neuroscience  in  terms  of  translating  fundamental  research  to  military 
relevant  domains;  that  is,  to  establish  and  articulate  fundamental  translational  principles.  These 
principles  are  envisioned  to  guide  the  development  of  technology  solutions  that  work  in  harmony 
with  the  capabilities  and  limitations  of  the  human  nervous  system  as  situated  within  its  dynamic, 
complex  environment. 

1.1  The  CaN  CTA 

The  purpose  of  this  report  is  to  present  the  overall  CaN  CTA  scientific  vision  and  approach,  as 
well  as  describe  the  three  initial  transitions  that  outline  paths  that  will  result  in  practical  solutions 
that  address  particular  Army  problems  or  unmet  needs. 

In  the  remainder  of  this  section,  we  briefly  describe  the  organization  and  organization  of  the  CaN 
CTA  program.  The  research  program  for  the  CaN  CTA  has  been  organized  into  three  technical 
areas  (TAs)  that  are  strongly  interconnected,  comprising  a  highly  integrated  research  and 
conceptual  approach. 
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TAl.  Neurocognitive  Performance:  Research  efforts  under  this  area  are  designed  to  ascertain 
key  “signatures”  of  how  Soldier  neurocognitive  state  (i.e.,  emotional,  perceptual-cognitive, 
physical,  and  physiological)  varies  in  the  face  of  the  sensory,  perceptual,  and  cognitive  demands 
of  the  operational  environment. 

TAl.  Advanced  Computational  Approaches:  Research  within  this  second  area  will  provide 
novel  computational,  statistical  modeling,  and  data  visualization  techniques  for  the  extraction  of 
the  signatures  of  Soldier  neurocognitive  performance,  including  novel  analytic/algorithmic 
approaches  to  the  individualized  assessment  of  Soldier  neurocognitive  state  and  performance. 

TA3.  Neurotechnologies:  The  third  area  focuses  on  the  research  underlying  cutting-edge 
technologies  for  the  real-time  recording  and  analysis  of  environmental,  behavioral,  and 
functional  brain  dynamics,  and  on  techniques  for  exploiting  natural  human  neurocognitive 
function  in  technological  solutions  that  enhance  Soldier-system  performance  and  safety. 

The  initial  membership  of  the  CaN  CTA  represents  a  truly  international  and  multidisciplinary 
team,  and  comprises  membership  from  the  following  industry  and  academic  institutions: 

1.  DCS  Corporation  (DCS),  Alexandria,  VA,  USA 

2.  University  of  California  San  Diego  (UCSD),  San  Diego,  CA,  USA 

3.  University  of  Michigan,  Ann  Arbor,  MI,  USA 

4.  University  of  Texas  San  Antonio  (UTSA),  San  Antonio,  TX,  USA 

5.  National  Chiao  Tung  University  (NCTU),  Hsinchu,  Taiwan 

6.  University  of  Osnabriick,  Osnabriick,  Germany 

Additionally,  the  CaN  CTA  is  pleased  to  welcome  three  new  collaborating  organizations,  as  well 
as  additional  research  partners  at  the  University  of  Michigan,  whose  research  efforts  as  sub¬ 
awardees  to  the  Alliance  will  bring  new  perspectives  and  approaches  to  addressing  the  critical 
challenges  being  addressed  by  CTA  researchers: 

1.  Columbia  University,  New  York,  NY,  USA 

2.  University  of  California  Santa  Barbara  (UCSB),  Santa  Barbara,  CA,  USA 

3.  University  of  Pittsburgh,  Pittsburgh,  PA,  USA 

A  listing  of  specific  projects  and  their  respective  principal  investigators  is  located  in  the 
appendix. 
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2.  Translational  Science 


2.1  Army  Relevance 

Future  Force  concepts  include  the  goal  of  establishing  and  maintaining  information  dominance 
by  implementing  advanced  information  and  communications  technologies  in  a  massively 
interconnected,  networked  battlefield,  including  an  increased  reliance  on  unmanned  assets  while, 
at  the  same  time,  decreasing  crew  sizes  to  minimize  the  number  of  Soldiers  in  harm’s  way 
(Speakes  and  Martin,  2008).  Attempting  to  preserve  operational  capabilities  while  decreasing 
local  crew  sizes  and  distributing  tasking  across  a  broad  network  of  non  co-located  teams  will 
require  the  future  combat  Soldier  to  adopt  a  significantly  modified  and  quite  demanding  role. 

For  example,  as  envisioned  under  Future  Force  concepts,  it  is  possible  that  an  individual  Soldier 
will  be  responsible  for  concurrent  monitoring  of  data  from  unattended  ground  sensors, 
maintenance  of  intra-  and  extra-vehicular  communications,  operation  of  non-line-of-sight 
munitions,  as  well  as  execution  of  direct,  remote  or  supervisory  control  over  a  vehicle  or  a 
number  of  other  unmanned  ground  and/or  aerial  assets  (Fetterman  and  Plushnik,  2007). 

What  is  more,  due  to  pervasive  threats  from  enemies  who  employ  unconventional  tactics,  such  as 
improvised  explosive  devices  (lEDs),  Army  operations  will  increasingly  be  conducted  either 
from  within  vehicles  that  are  completely  enclosed  in  armor  or  from  locations  remote  to  the 
battlefront.  Therefore,  both  mounted  and  dismounted  Soldiers  will  execute  most  tasks  involving 
interactions  with  the  battlefield  through  a  variety  of  computerized  control  and  visualization 
systems  (Keller,  2004),  rather  than  based  on  information  from  first-person  immersion  in  the 
immediate  zone  of  conflict. 

The  fact  that  such  operations  will  inevitably  be  conducted  within  a  dynamic,  multidimensional 
urban  context,  where  battles  tend  to  unfold  on  shorter  temporal  and  spatial  scales,  only  serves  to 
increase  the  level  of  demand,  threat  and  risk  to  U.S.  troops.  Because  of  limitations  on  both 
technology  and  human  function  within  such  complex  operational  environments,  performance 
decrements  are  a  chief  concern  of  those  examining  issues  related  to  the  development  and 
implementation  of  technologies  that  can  enhance  the  function  of  integrated,  human-machine, 
sociotechnical  systems  (Cowings,  Toscano,  DeRoshia,  and  Tauson,  2001;  Endsley  and  Kaber, 
1999;  McDowell,  Oie,  Tierney,  and  Elascher,  2007;  Metcalfe,  Davis,  Tauson,  and  McDowell, 
2008;  Stanton  and  Marsden,  1997;  Stanton  and  Young,  1998;  and  van  Erp  and  Padmos,  2003). 

The  skilled  cognitive  and  sensorimotor  performance  that  underlies  effective  mission  use  of 
human-machine,  sociotechnical  systems  is  clearly  organized  at  the  level  of  the  nervous  system. 
Thus,  in  order  to  develop  neuroscience-based  systems  to  meet  critical  Army  needs,  the  CaN  CTA 
program  has  been  designed  to  provide  the  systematic  exploration  of  the  neural  underpinnings  of 


5 


Soldier-system  performance  needed  to  elucidate  new  understandings  of  human  neurocognitive 
function  as  situated  within  the  complex,  dynamic  task  environments  that  typify  Army  operations. 

The  advances  to  be  gained  from  the  CaN  CTA  program  hold  the  potential  for  radical  and 
revolutionary  changes  to  systems  design  and  development  to  enhance  Soldier-system 
performance:  The  improved  understandings  of  Soldier  neurocognitive  state  and  performance 
within  the  operational  environment  that  are  the  aim  of  the  Neurocognitive  Performance  research 
area  will  provide  the  fundamental  knowledge  required  to  enable  a  systems  design  approach  that 
can  work  in  harmony  with  human  brain  function.  The  computational  models  and  data  analytic 
methods  developed  under  the  Advanced  Computational  Approaches  research  area  will  provide 
Army  researchers  and  engineers  not  only  with  new  approaches  to  understanding,  but  also  with 
the  tools  to  develop  automated  systems  and  intelligent  architectures  that  integrate  real-time 
knowledge  of  Soldier  neurocognitive  state  to  complement  and  supplement  Soldier  abilities. 
Moreover,  the  technologies  and  tools  for  functional  brain  imaging-based  detection  of  Soldier 
cognitive  performance  that  will  be  developed  through  the  third  Neurotechnologies  research  area, 
as  well  as  planned  CaN  CTA  transition  efforts,  have  far-reaching  translational  potential,  not  only 
within  Army  and  Joint  service  materiel,  training,  medical,  and  testing  domains,  but  also 
throughout  the  civilian  science  and  technology  (S&T)  community. 

Finally,  the  overarching  aim  of  the  research  to  be  conducted  by  the  CaN  CTA  is  to  develop  and 
demonstrate  principles  that  govern  the  translation  of  research  in  cognitive  neuroscience  from 
basic  to  applied  levels  and  that  can  guide  efforts  to  best  support  future  Soldier-system 
technological  development.  Such  principles  must  be  both  grounded  in  data  and  theory  from 
basic  cognitive  neuroscience  research  and  reflective  of  a  deep  understanding  of  how  the  human 
brain,  body,  and  sensory  systems  work  in  concert  to  accomplish  tasks  within  militarily-relevant 
operational  contexts.  Once  established,  these  translational  principles  will  be  fundamental  in 
driving  and  speeding  the  transition  of  Army  basic  and  applied  research  efforts  into  high-impact, 
mission-enabling  technologies. 

2.2  Scientific  Barriers  and  Hurdles 

The  main  problem  that  the  CaN  CTA  is  addressing  is  that  mission-enabling  technologies  and  the 
methods  for  technological  advancement  do  not  adequately  account  for  the  capabilities  and  the 
limitations  of  human  neurocognitive  function.  In  large  part,  this  state  of  affairs  can  be  attributed 
to  the  S&T  community  lacking  the  theory  and  tools  required  to  provide  a  clear  working 
understanding  of  how  the  human  brain  functions  when  faced  with  complex  real-world  tasks  in 
real-world  operational  settings,  as  opposed  to  how  it  works  in  highly  controlled  experiments  in 
highly  constrained  laboratory  settings.  To  fully  understand  Soldier  neurocognitive  performance 
in  a  manner  that  can  support  a  neuroscience-based  systems  design  approach,  it  is  critical  to 
directly  study  and  precisely  quantify  the  natural  interactions  among  Soldier  brain,  behavioral, 
sensory,  and  performance  dynamics  as  situated  within  complex  task  environments. 
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Five  conceptual  and  technological  limitations  have  thus  far  precluded  our  capability  to  leverage 
advanced  technologies  to  address  the  lack  of  knowledge  regarding  human  brain  function  as 
situated  in  the  real  world. 

B.lThe  restriction  of  experimental  designs  to  highly  controlled  and  impoverished 
stimulus/response  paradigms  and  environments. 

Both  human  behavior  and  the  environment  within  which  it  unfolds  are  rich,  dynamic,  and 
complex.  To  harness  the  complex  dynamics  of  environment  and  behavior,  the  scientific  study  of 
human  performance  has  traditionally  employed  paradigms  that  aimed  at  simplicity  in  both 
stimuli  and  measured  behavior.  The  vast  cacophony  of  sounds  that  makes  up  the  auditory 
environment,  for  example,  has  typically  been  reduced  to  single  frequency,  limited  duration 
“beeps,”  while  the  rich,  continuous  array  of  visual  contours,  colors,  and  contrasts  present  in  the 
natural  visual  environment  has  typically  been  replaced  in  laboratory  experiments  with  spatially, 
temporally  isolated  simple  patterns,  gradients,  or  light  flashes  with  stereotyped  and  tightly 
controlled  properties  (contrast,  luminance,  color,  etc.).  Motor  behavior,  in  like  fashion,  has  often 
been  studied  through  single  joint  (e.g.,  button  press),  single  limb  (e.g.,  point  and  reach),  or 
simplified  whole-body  (e.g.,  standing)  actions.  However,  as  discussed  above,  to  understand 
natural  behavior,  it  is  critically  important  to  migrate  research  from  such  controlled  tasks  and 
environments  into  those  that  reflect  the  complex  interactions  between  natural  behavior  and  the 
natural  sensory  world  within  which  it  is  embedded.  Perhaps  the  most  potent  evidence  for  such  a 
position  is  to  be  found  in  arguments  articulating  a  fundamental  disconnect  between  observations 
in  laboratory  settings  and  those  obtained  from  within  a  real-world  context  (Kingstone  et  ah, 
2008);  that  is,  what  is  seen  in  the  laboratory  may  not  (and  often  does  not)  necessarily  represent 
how  the  brain  actually  functions  when  presented  with  a  real  task  and  situated  in  a  real-world 
setting  (Foerde  et  ah,  2006). 

B.lThe  lack  of  portable,  user-acceptable  (e.g.,  comfortably  wearable),  and  robust  systems 
for  monitoring  brain  and  body  dynamics. 

If  we  aim  to  observe,  in  a  precisely  quantifiable  fashion,  humans  performing  realistic  tasks  as 
naturally  as  possible  within  operationally  relevant  contexts,  then  improvements  must  be  made  to 
current  methods  of  instrumentation.  That  is,  in  order  to  allow  for  reasonable  inference  regarding 
natural  behavior  in  such  contexts,  human  participants  must  be  allowed  to  behave  as  they  would, 
without  being  encumbered  by  extensive  arrays  of  sensors,  leads,  power  supplies,  and  processing 
units.  At  the  same  time,  a  considerable  array  of  sensors  is  necessary  to  capture  sufficiently 
precise  and  detailed  information  regarding  brain  and  behavioral  dynamics,  especially  to  avoid 
confounds  due  to  the  behaviors  being  performed  as  well  as  environmental  fluctuations  that  may 
introduce  variable  levels  of  artifact  into  the  data.  Thus,  an  ideal  monitoring  system  would  be 
comfortably  wearable  for  participants  as  they  perform  natural  behavior  and  yet  would  also 
contain  a  complete  and  robust  array  of  sensors  to  provide  the  detail  necessary  for  understanding 
true,  contextually  situated  behavior. 
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B.3The  failure  to  record  the  whole  of  physical,  mental,  and  physiological  behavior  that  the 
brain  controls,  and  the  physical  and  socio-cultural  effects  of  the  environment  that  impact 
brain  function,  in  sufficient  detail  and  across  a  sufficient  breadth  of  circumstances. 

The  evolution  of  the  human  brain  has  been  shaped  by  the  need  to  adaptively  optimize  the 
outcome  of  our  behavior  within  the  context  of  our  three-dimensional  (3-D)  environment. 

Imaging  studies  show  that  human  brain  systems  that  organize  our  decision  making  involve  the 
same  systems  as  those  that  organize  our  motor  behavior  (Rizzolatti  et  ah,  2002).  Likewise,  the 
ways  we  think  about  and  communicate  abstract  concepts  are  often  firmly  based  in  our  bodily 
experience  in  the  3-D  world  (e.g.,  “Let’s  get  in  touch.”. . .  ‘Time  marches  on.”. . .).  The 
conceptual  meaning  of  this  argument  is  that  brain  function  cannot  be  understood  in  isolation 
from  the  behavior  that  is  to  be  controlled  and  the  context  within  which  it  is  to  be  controlled.  Yet 
the  traditional  enterprise  of  science  has  followed  a  course  that  has  led  to  a  fractionation  of  our 
understanding  of  the  whole  of  human  behavior  into  relatively  disjoint  pieces  falling  within  the 
respective  purviews  of  individual  academic  domains.  Specific  brain  function  (e.g.,  the  behavior 
of  neurons  and  neural  circuits),  for  example,  has  fallen  under  the  large  umbrella  of  neuroscience, 
while  emergent  properties  of  neural  function  (e.g.,  cognition,  perception,  etc.)  have  traditionally 
been  studied  within  the  cognitive  and  psychological  sciences  and,  finally,  physical  actions 
themselves  have  been  the  subject  of  movement  science  and  biomechanics.  Although  broader 
consideration  of  the  whole  ensemble  of  behavior,  particularly  relations  between  motor  and 
cognitive  aspects,  has  begun  to  appear  more  frequently  in  the  literature  (c.f.  Middleton  and 
Strick,  2000;  Rosenbaum,  2005),  it  remains  a  fundamental  challenge  to  build  the  database  of 
collective  knowledge  regarding  how  the  brain  works  while  it  is  actively  organizing  and 
executing  cognitive  and  physical  action  within  the  natural  environment  and  across  a  set  of  tasks 
representative  of  the  natural  functional  repertoire  of  the  skilled  human  making  strategic  decisions 
based  on  accumulating  streams  of  natural  sensory  data. 

B.4The  lack  of  mathematical  modeling  methods  and  software  to  find  statistical  relationships 
between  moment-to-moment  variations  in  environmental,  behavioral,  and  functional 
brain  dynamic  recordings. 

Beyond  the  acquisition  of  data  sufficient  to  draw  inferences  regarding  natural  human  behavior 
within  rich,  real-world  settings  is  the  need  for  analytic  tools  to  detect  and  identify  patterns  and 
relationships  within  those  data.  For  example,  the  focus  of  most  traditional  analyses  in 
psychology,  biodynamics,  and  neuroscience  has  been  on  modeling  repeatable  and  (often)  passive 
interactions  allowing  the  use  of  simple  response  averaging  methods  and  second-order  statistics. 
Currently,  however,  there  are  increasing  needs  to  model  nonlinear  interactions  in  biological  and 
cognitive  systems,  and  develop  methods  to  identify  complex  patterns  in  huge  datasets  using  new 
information-based  statistical  methods.  Consider  Soldiers  and  commanders  faced  with  unfolding 
challenges  in  complex  operational  environments  who  need  to  make  operational  decisions  by 
interacting  with  multiple  streams  of  incomplete  and  possibly  misleading  information.  Optimum 
models  of  their  learning  and  decision-making  processes  cannot  simply  combine  separate 
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psychological,  biodynamic,  and  neurodynamic  models,  as  these  all  interact  in  complex  (often 
nonlinear)  ways.  True  understanding  of  human  behavioral  decision  making  under  stress  in 
complex  operational  environments,  therefore,  requires  direct,  quantitative,  and  model-based 
study  of  brain,  behavioral,  sensory,  and  performance  dynamics,  based  on  their  simultaneous 
measurement  in  complex,  operationally  relevant  environments  and  their  joint  analysis  accounting 
for  their  higher-order  interactions  and  relations. 

B.SThe  lack  of  sufficient  data  archives  and  resources  to  systematically  study  relationships 
between  individualized  models  derived  for  cognitive  monitoring  and  individual 
differences  in  performance,  cognitive  ability,  and  personality,  relationships  that  could 
optimize  performance  of  cognitive  monitoring  systems  while  lowering  requirements  for 
collecting  individual  subject  training  data. 

Typical  studies  of  human  cognitive  function  using  brain  imaging  collect  data  from  10-40 
subjects  with  goals  of  (1)  determining  a  population  mean  and  variance,  the  variability  in  the 
measures  of  interest  thereby  assumed  to  be  distributed  in  a  Gaussian  manner  across  the  subject 
population,  or  (2)  determining  the  difference  between  two  such  subject  group  means  with,  most 
often,  Gaussianity  and  equal  variances  assumed  for  both  groups.  However,  increasingly 
cognitive,  brain  morphogenic,  and  neurogenetic  studies  reveal  fine  differences  in  cognitive 
abilities  and  style  that  limit  overall  performance  of  human-machine  systems  that  do  not  adapt 
themselves  to  individual  operator  proclivities  and  characteristics.  Development  of  principles 
governing  construction  of  individually  adapted  human-system  interfaces  is  the  principal  aim  of 
the  CaN  CTA. 

2.3  The  CaN  CTA  Scientific  Vision 

Across  the  broad  spectrum  of  Army  operations.  Soldiers  and  commanders  are  expected  to 
continually  make  decisions  based  on  distributions  of  information  across  various  temporal  and 
spatial  scales  that  they  acquire  by  active  attention  to  and  interaction  with  multiple  audio,  video, 
and/or  tactile  information  streams.  Successful  decision  making  in  such  environments  requires 
operator  recognition  of  the  overall  significance  of  accumulating  information  so  as  to 
continuously  maintain  situational  awareness  and  answer  operationally  relevant  command 
questions  posed  by  events.  Enabling  optimized  decision  making  defined  in  this  manner  requires 
an  understanding  of  the  intimate  relationships  between  the  Soldier  and  the  operational 
environment  as  mediated  by  physiology,  cognition,  and  overt  behavior. 

We  posit  that  applying  basic  research  in  cognitive  neuroscience  to  understanding  Soldier 
neurocognitive  behavior  within  these  dynamic,  complex  environments  will  be  most  fruitful  if  it 
can  continually  and  successfully  monitor  and  interpret  brain/behavioral  processes,  indexing 
(1)  the  depth,  distribution,  and  shifting  of  operator  attention;  (2)  the  operator’s  appraisal  of  the 
significance  of  incoming  information;  (3)  the  emotional  context  (e.g.,  motivation  and  intent)  of 
the  operator’s  actions;  and  (4)  the  impact  of  physiological  state  (e.g.,  fatigue,  stress,  arousal)  on 
cognitive  and  motor  performance.  In  turn,  such  understandings  are  required  to  develop  systems 
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that  can  exploit  the  natural  properties  of  human  neurocognitive  and  behavioral  function  to 
compliment  and  reinforce  both  individual  and  group  operator  abilities  under  dynamic  and  high 
information  loads,  as  well  as  under  circumstances  of  multiple  objectives  with  differing  levels  of 
risk  and  reward.  In  other  words,  these  understandings  are  needed  to  enable  the  design  and 
development  of  systems  whose  ultimate  purpose  is  to  enhance  Soldier-system  capabilities  to 
more  effectively  meet  mission-critical  demands. 

Objectives  such  as  these  make  observing  and  modeling  how  distributed  brain  processes  support 
natural,  active,  and  ever-varying  behavior  and  cognition  perhaps  the  most  important  challenge  to 
successful  translation  of  principles  and  methods  from  cognitive  neuroscience.  As  articulated  in 
ecological  psychology  and,  in  particular,  in  Gibson’s  (1979)  Theory  of  Ajfordances,  humans  are 
active  agents,  continually  engaged  in  actively  attempting  to  fulfill  their  needs  and  desires  within 
a  complex  and  dynamic  environment,  often  in  concert  and/or  competition  with  other  human, 
animal,  and  mechanical  agents.  Such  influences  and  constraints  have  served  to  shape  the 
evolution  of  the  human  brain  by  means  of  optimizing  the  outcome  of  our  behavior. 

At  the  same  time,  concepts  tied  together  under  the  terms  “embodied  cognition”  and/or  “situated 
cognition”  have  become  increasingly  important  in  cognitive  science  and  neuroscience.  In  a 
similar  fashion,  these  concepts  have  been  formed  around  the  view  that  cognitive  processes  have 
evolved  to  optimize  the  outcome  of  our  body-based  behavior  within  our  3-D  environment 
(Anderson,  2003;  Clark,  1999;  Garbarini  and  Adenzato,  2004).  Key  principles,  or  assumptions, 
include  the  following: 

•  Perception  and  cognition  support  action.  A  primary  function  of  human  perception  is  to 
assist  motor  control  (Churchland  et  ah,  1994;  Reed,  1982).  For  example,  the  dorsal  visual 
pathway  directly  supports  visually  guided  actions  such  as  grasping  (Goodale  and  Milner, 
1992).  In  monkeys  and  humans,  motor  neurons  involved  in  controlling  tools  are  active 
when  pictures  of  tool  use  are  viewed,  even  in  the  absence  of  overt  motor  actions  or 
planning  (Rizzolatti  and  Arbib,  1998). 

•  Cognition  is  situated  and  time-pressured.  New  information  is  integrated  into  our 
continually  evolving  cognitive  environment,  allowing  us  to  adapt  to  and  predict  how  events 
in  our  current  environment  may  be  influenced  by  our  actions.  Action-motivating  events 
typically  require  timely,  environment-  and  situation-appropriate  action  selection. 

•  VFe  actively  interact  with  our  environment  to  gain  and  hold  situational  awareness.  Our 
typical  feeling  that  we  “can  see  everything”  around  us  rests  on  the  feeling  that  what  we  see 
gives  us  enough  information  to  know  where  to  actively  look  for  further  information,  when 
and  if  it  becomes  of  interest.  To  further  reduce  our  brain  memory  and  processing  load,  we 
actively  manipulate  our  environment. 
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•  Features  of  our  environment  are  integrated  into  our  cognitive  system.  Our  brain’s  body 
image  extends  beyond  our  physical  body  (Maravita  and  Iriki,  2004).  For  example,  a  tennis 
player’s  racket  becomes  an  integral  part  of  the  player’s  brain/body  “action  system”  (Reed, 
1982),  influencing  the  environment  (the  ball)  to  shape  future  events  (the  opponent’s 
return). 

•  Our  imagination  and  abstract  cognition  are  also  bodily  experience  based.  Even 
imaginative  and  abstract  cognition,  independent  of  direct  physical  interaction  with  the 
environment,  are  body-based.  For  instance,  sub-vocal  rehearsal  of  contents  in  verbal 
working  memory  involves  the  same  brain  structures  used  for  speech  perception  and 
production  (Wilson,  2001),  and  imagining  limb  movements  produces  activity  in  the  same 
brain  areas  involved  in  producing  the  actual  movements.  We  interpret  “abstract”  linguistic 
metaphors  and  even  abstract  mathematical  entities  within  (“as-if ’)  virtual  environments, 
such  as  when  we  “look  forward”  to  the  future,  or  imagine  “the  number  line”  (Nunez,  2006). 
Emotions  may  similarly  involve  somatic  “as-if’  states  and  events  (Damasio  et  ah,  2000). 

Acceptance  of  these  principles  imposes  particular  constraints  on  the  methodological  decisions 
one  must  make  when  attempting  to  understand  natural  behavior  in  realistic  environments.  In 
particular,  such  concepts  dictate  that  it  is  literally  impossible  to  understand  natural,  motivated 
behavior  in  artificial  scenarios  and  situations  that  divorce  the  individual  from  the  context.  In 
short,  the  reductionist  paradigms  that  have  thus  far  defined  the  “normal  science”  behind  studies 
of  human  behavior  provide  information  that  is  impoverished  to  the  extent  that  the  true  nature  of 
human  performance  appears  intractable.  Indeed,  recent  findings  have  demonstrated  the 
importance  of  context  in  the  differential  engagement  of  neural  resources  during  task  performance 
(Foerde  et  ah,  2006),  supporting  the  idea  that  laboratory-based  studies  isolated  from  real-world 
experience  may  generate  fundamental  misunderstandings  of  the  underlying  principles  being 
investigated  (cf.,  Kingstone  et  ah,  2008).  Therefore,  it  is  incumbent  upon  cognitive  neuroscience 
to  more  adequately  address  the  multiple  hitherto  unanswered  questions  of  how  distributed  brain 
dynamics  support  complex,  motivated  actions  within  dynamic  and  only  partially  predictable  real- 
world  operational  contexts. 

True  understanding  of  human  behavioral  decision  making  under  stress  in  complex  operational 
environments,  therefore,  requires  direct  study  of  the  interactions  among  brain,  behavioral, 
sensory,  and  performance  dynamics  in  the  context  of  task  and  environmental  dynamics,  based  on 
their  simultaneous  measurement  and  joint  analysis.  In  other  words,  cognitive  neuroscience  must 
develop  new  approaches  to  augment  traditional  ones  that  have  largely  divorced  human  behavior 
from  the  richness  of  its  physical  instantiation  and  context  in  order  to  represent  it  through  the 
activity  of  a  few  neural  populations  in  the  cerebral  cortex.  These  new  approaches — both  in  terms 
of  experimental  paradigms  and  data  analytic  methodologies — must,  as  much  as  possible,  mine 
the  full  spectrum  of  information  available  throughout  the  natural  behavioral  context  for  insights 
into  fundamental  neural  principles  that  guide  the  organization  of  motivated  action  in  real-world 
environments. 
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3.  The  CaN  CTA  Approach  to  Translational  Science 


Providing  solutions  to  enhance  neurocognitive  human-system  performance  demands  new 
approaches  to  basic  science.  Here  we  outline  such  an  approach  that  identifies  and  develops 
fundamental  principles  translating  cognitive  neuroscience-based  theory  and  observations  to 
enabling  technologies  that  enhance  performance  in  militarily -relev ant  environments. 

Specifically,  the  CaN  CTA  research  program  aims  to  develop  neuroscience-based  principles  that 
focus  on  how  best  to  assist  people  to  operate  in  complex,  mixed-initiative  system  environments 
in  which  dynamic,  complex  information  streams  must  be  integrated  across  multiple  spatial  and 
temporal  scales.  The  CaN  CTA  approach  focuses  on  two  factors:  first,  we  directly  address  the 
scientific  barriers  outlined  in  section  2.2  to  enable  translational  science;  and  second,  we  execute 
translational  science  within  specific  neurocognitive  performance  domains. 

3.1  Addressing  Scientific  Barriers  to  Enable  Translational  Science 

The  CaN  CTA  will  adopt  an  approach  that  directly  addresses  the  five  scientific  and  technical 
challenges  identified  in  section  2.2. 

B.lAddressing  the  restriction  of  experimental  designs  to  highly  controlled  and  impoverished 
stimulus/response  paradigms  and  environments. 

Despite  the  advances  obtained  through  the  use  of  functional  neuroimaging  modalities  over  the 
past  few  decades,  the  physical  and  temporal  constraints  of  technologies  such  as  functional 
magnetic  resonance  imaging  (fMRI)  and  positron  emission  tomography  (PET)  have  severely 
limited  the  scope  of  brain  imaging  during  production  of  naturally  motivated  motor  behavior,  e.g., 
behavior  involving  whole  body  motion  in  normal  3-D  environments.  Thus,  the  use  of  such 
methods  has  forced  investigators  to  observe  neural  function  separate  from  its  natural  behavioral 
context. 

It  is  our  position  that,  currently,  electroencephalography  (EEG)  is  the  only  brain  imaging 
modality  that  involves  sensors  light  enough  and  low-powered  enough  to  allow  near-complete 
freedom  of  movement  of  the  head  and  body  with  sufficient  time  resolution  to  record  brain 
activity  on  the  time  scale  of  natural  behavior.  Therefore,  EEG  is  the  clear  modality  of  choice  for 
brain  imaging  of  humans  performing  tasks  involving  natural  movements  in  real-world 
operational  environments. 

Unfortunately,  because  of  how  EEG  has  traditionally  been  used  to  study  neurocognitive  function, 
limitations  in  its  method  of  analysis  has  forced  investigators  to  make  trade-offs  in  terms  of  the 
fidelity  and  nature  of  behavior  observed.  In  particular,  traditional  EEG  paradigms  severely 
restrict  body,  head,  and  eye  movements  of  participants — largely  out  of  fear  of  introducing  non¬ 
brain  artifacts  into  the  EEG  data  records.  In  traditional  EEG  experiments,  researchers  have 
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typically  measured  only  minimal  participant  behaviors;  usually  stereotyped  button  press 
responses  to  a  limited  set  of  discrete,  temporally  distinct,  and  suddenly  presented  stimuli. 

Because  of  the  perceived  difficulty  of  separating  brain  EEG  data  from  non-brain  artifacts, 
participants  in  such  EEG  experiments  have  been  asked  to  sit  still,  suppressing  or  minimizing 
natural  eye  and  head  movements,  waiting  for  stimulus  onsets  and  then  making  relatively 
infrequent  responses  involving  minimal  limb  movements  (typically,  pressing  finger 
“microswitches”)  to  convey  their  selections  from  among  a  discrete,  limited,  and  artificial  range 
of  choices. 

In  well-established  psychophysiology  paradigms,  simple  EEG  response  averages  that  are  time- 
locked  to  events  in  a  number  of  a  priori  defined  categories  are  then  extracted  from  the  brain  data, 
reducing  its  complexity  to  a  few  averaged  traces  that  are  further  collapsed  into  an  even  smaller 
table  of  peak  amplitudes  and  latencies.  Researchers  then  look  for  reliable  relationships  between 
these  few  summary  values  and  a  single  behavioral  dependent  variable  (e.g.,  the  button  the 
participant  chose  to  press  in  each  trial). 

This  approach  essentially  attempts  to  reduce  the  complexity  of  the  recorded  EEG  dynamics  (now 
easily  recorded  with  a  bandwidth  of  a  million  or  more  bits  per  second)  to  near  the  bandwidth  of 
the  recorded  behavior  (typically  less  than  one  button  selection  per  second)  by  averaging  across 
epochs  time-locked  to  sets  of  events  assumed  to  have  similar  brain  dynamic  consequences.  Yet, 
given  the  complexity  and  marked  moment- to-moment  variability  of  human  EEG  dynamics,  as 
well  as  the  brain’s  central  role  in  optimizing  the  outcome  of  behavior  in  the  face  of  ever- 
changing  physical  and  cognitive  circumstances,  it  is  unlikely  that  this  reductionist,  a  priori 
approach  will  yield  further  high-payoff  advances  in  understanding  how  distributed  brain 
dynamics  support  natural  operator  behavior  and  processing  of  real-world  operational 
environments. 

Erom  a  mathematical  point  of  view,  then,  the  basic  problem  outlined  above  is  that  complex 
functional  relationships  between  two  or  more  high-dimensional  and  highly  variable  signals  (e.g., 
brain  dynamics  and  overt  behavior)  cannot  be  well  characterized  by  constraining  each  to  a  few 
average  measures  and  then  comparing  them.  Rather,  we  argue  that  what  is  needed  is  a  new  and 
radically  different  approach  incorporating  better,  higher  fidelity  recording  and  modeling  of  the 
full  spectrum  of  information  regarding  relationships  between  neural  and  physiologic  function, 
cognition,  and  overt  behavior. 

The  approach  being  taken  within  the  CaN  CTA  program  is,  therefore,  to  combine  data  from 
multiple,  data-intensive  sources  so  that  the  information  latent  in  the  observed  brain/body/system 
dynamics  may  be  identified  as  described  above.  The  CaN  CTA  research  then  intends  to  develop 
novel  computational  and  mathematical  modeling  methods  to  discover  functional  and  statistically 
reliable  relationships  between  simultaneously  recorded  brain,  behavioral,  and  sensory  dynamics. 
Important  issues  to  be  addressed  are  how  to  extract  information  that  is  invariant,  stable,  and 
robust  to  changes  and/or  degradation  of  the  monitoring  environment.  Eurther,  we  want  to  be 
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proactive  in  enhancing  the  operator’s  cognitive  and  perceptive  abilities,  not  reactive  after  the 
fact.  For  example,  we  want  to  predict  when  a  human  operator  is  likely  to  experience  cognitive 
overload  or  incapacity  prior  to  or  at  time  of  its  first  occurrence.  Also,  we  would  like  to  identify 
immediately  when  an  operator  has  hastily  made  a  command  decision  (e.g.,  by  button  press)  that 
the  operator  now  regrets. 

To  approach  the  richness  and  multidimensional  variety  of  Soldier  experience  in  relevant 
operational  environments,  Simulation  and  Demonstration  phase  experiment  tasks  will  be 
complex.  In  particular,  they  will  require  operators  to  integrate  information  across  multiple 
streams  in  different  modalities.  Operators  will  thus  be  required  to  continually  ask  themselves, 
“To  which  information  channel  should  I  attend  now  to  best  maintain  my  awareness  of  the 
operational  situation  ?  Should  I  actively  query  or  redirect  any  information  channels  ?  Do  I  have 
information  I  should  I  send  my  commanders  and/or  remote  team  members  ?  Do  I  need  to  make  a 
command  decision  to  direct  remote  team  members?  If  so,  what  decision?” 

The  main  Command  Environment  Simulator  (CES)  experiment  to  be  conducted  at  the  of  UCSD 
may,  for  instance,  involve  an  extended  operational  game  space  in  which  ally,  foe,  and  indifferent 
agents  will  move  and  act.  Eor  example,  in  an  urban  conflict  simulation,  unnumbered, 
autonomous  competitive  agents  (foes)  may  enter  the  3-D  game  space  (an  urban  neighborhood)  at 
different  rates  of  speed  under  stochastic  control.  Numbered,  semi-autonomous  cooperative 
agents  (allies)  respond  to  the  commands  of  the  operator  stationed  in  a  home  vehicle  command 
post.  The  operator  may  decide  to  order  an  ally  to  move  so  as  to  neutralize  the  advance  of  a  foe. 
The  operator  will  see  displays  showing  simulated  views  from 

(1)  video  cameras  on  the  vehicle;  (2)  a  (slowly)  movable  robot  video  camera;  and  (3)  a  (slowly) 
movable  overhead  still  camera  drone.  The  operator  will  receive  speech  communications  from 
allies  and  occasionally  from  an  off-map  commander. 

In  the  seenario  thus  deseribed,  the  operator’s  task  would  be  to  position  allies  so  as  to  neutralize 
threats  from  moving  foes.  The  visual  displays  (vehicle,  robot,  drone)  would  each  give  spatially 
limited  egocentric  and  allocentric  views  of  the  operational  space  on  different  spatial  (distance) 
and  temporal  (update  rate)  scales.  Speech  communications  from  allies  about  foe  locations  and 
movements  would  either  be  given  either  from  an  egocentric  perspective  (“Eoe  approaching 
quickly”)  or  an  allocentric  perspective  (“Eoe  moving  west  slowly”),  and  would  sometimes  be 
misleading  or  inaccurate.  Eocation-based  speech  communications  would  be  egocentric  (“I  am 
two  blocks  north  and  one  block  east  of  you,  facing  west”).  Spatial  auditory  and  vibrotactile 
reinforcement  might  be  used,  if  shown  effective  in  earlier  experiments.  All  experimental  events 
would  thus  be  recorded  along  with  brain/body  imaging  data  from  freely  moving  experiment 
participants  to  create  a  large  database  for  studying  relationships  between  operator  attention- 
shifting  and  decision-making  success,  and  model  neurobehavioral  differences  between  subjects. 
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B.lAddressing  the  lack  of  portable,  user-acceptable  (e.g.,  comfortably  wearable),  and  robust 
systems  for  monitoring  brain  and  body  dynamics. 

Portable  high-density  EEG  systems  are  already  available  to  study  the  EEG  dynamics 
accompanying  motivated  actions  in  normal  3-D  environments.  However,  a  long-missing  link 
between  advancing  the  understanding  of  EEG  signals  and  practical  workplace  applications  has 
been  the  need  to  use  wet  gel  to  adhere  EEG  electrodes  to  the  scalp  and  the  need  to  bring  scalp 
impedance  of  these  contacts  down  to  near  5  kG  by  abrading  the  scalp  with  a  blunt  needle  or 
equivalent  before  each  electrode  application. 

To  enable  high-quality  EEG  monitoring  of  naturally  moving  subjects,  EEG  systems  must  have 
characteristics  not  available  in  current  brain  imaging  systems.  To  support  dense  spatial 
sampling,  EEG  sensors  must  be  small  and  lightweight  and  not  require  special  skin  preparation. 
Eurther,  the  ideal  system  should  avoid  the  risk  of  electrical  bridging  between  nearby  contacts  by 
avoiding  the  use  of  conductive  gel.  To  minimize  weight  and  susceptibility  to  system  movement 
artifacts,  EEG  acquisition  and  amplification  circuits  must  also  be  small,  lightweight,  and  battery- 
powered.  To  maximize  mobility  and  allow  near  real-time  use  of  the  recorded  data  or  measures 
derived  from  it,  they  may  use  wireless  telemetry.  Continuing  research  into  microelectronic  bio¬ 
sensors  has  led  to  several  dry  electrode  designs  (Alizadeh  Taheri  et  ah,  1996;  Gondran  et  ah, 
1995;  Griss  et  ah,  2001),  making  truly  mobile,  wireless  EEG  systems  incorporating  dry  sensor 
technology  and  small,  lightweight,  wearable  data  acquisition  circuits  now  feasible  and  available 
for  current  experiments. 

Our  Alliance  includes  a  team  at  NCTU,  Taiwan  (C-T  Ein,  lead),  that  is  designing,  building,  and 
testing  light,  wearable,  wireless,  low-cost,  high-density,  microelectronic  EEG  systems  with 
successively  higher  sensor  densities.  NCTU  investigators,  along  with  UCSD  research  partners, 
have  recently  demonstrated  the  feasibility  of  using  dry  microelectromechanical-system  (MEMS) 
EEG  sensors  and  miniaturized  supporting  hardware/software  to  continuously  collect  EEG  data 
from  four  frontal  non-hairy  scalp  sites  in  a  realistic  virtual-reality-based  (VR)  driving  simulator 
(Ein  et  ah,  2008)  shown  in  figure  1.  In  the  study  of  Ein  et  al.  (2008),  a  signal-processing  module 
was  programmed  to  assess  fluctuations  in  individuals’  alertness  and  capacity  for  cognitive 
performance — accurate  steering  in  a  realistic  driving  simulator  (left  center) — based  on  EEG 
signals  measured  from  the  forehead  by  4-8  dry  electrodes  embedded  in  the  frontal  brim  of  a 
baseball  cap.  The  system  also  delivers  arousing  feedback  to  the  driver  (upper  right)  to  maintain 
performance.  This  platform  can  be  programmed  to  function  in  ways  appropriate  for  other  brain- 
system  interface  applications  (Ein  et  ah,  2008). 
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Figure  1 .  A  wearable  and  wireless  dry-electrode  (WWD)  EEG  system  that  features  dry 
MEMS  EEG  sensors  (lower  left),  low -power  signal  acquisition,  amplification 
and  digitization,  wireless  telemetry  (upper  left),  with  online  signal  processing. 

Despite  sueh  suecesses,  more  development  and  testing  are  needed  to  miniaturize  the  supporting 
hardware  so  as  to  eollect  high-density  EEG  data  (64  ehannels  or  more)  robustly  and  eontinuously 
from  unconstrained,  freely  moving  subjects  in  everyday  environments.  One  of  the  biggest 
challenges  to  the  current  MEMS -based  dry  electrodes  is  the  ability  to  acquire  EEG  signals  from 
hairy  scalp,  as  the  current  NCTU  MEMS  electrode  is  too  short  to  “penetrate”  the  hair  and  make  a 
good  skin  contact.  The  NCTU  team  has  recently  explored  a  potential  remedy  to  this  problem 
using  a  small  dry  electrode  array  with  longer  contact  “fingers”  (figure  2).  The  NCTU  team  is 
also  developing  a  flexible  electrode-supporting  polymer  substrate  to  construct  the  electrode 
“fingers.”  This  electrode  is  constructed  of  metal  embedded  with  polymer  and  coated  by  bio¬ 
friendly  silver  chloride  (AgCl).  Preliminary  results  have  shown  a  strong  correlation  (r  >0.92) 
between  the  signals  acquired  by  adjacent  conventional  wet  and  the  new  dry  electrodes  placed  at 
adjacent  scalp  sites  through  the  hair,  indicating  strong  potential  for  the  future  development  of 
EEG  technologies  that  can  support  truly  fieldable  neuroergonomic  systems  designs. 
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Figure  2.  Through-the-hair  dry  electrode  design.  A  yet  newer  (07/09), 
low-cost  ($0.50U.S.)  design  (not  shown)  has  “fingers”  that 
make  varying  height  and  angle  contacts  with  the  scalp. 

As  with  high-fidelity  recording  of  brain  activity,  recording  and  analysis  of  body  dynamics,  eye 
movements,  and  physiological  function  is  becoming  progressively  miniaturized  for  wearability. 
Methods  for  capturing  unconstrained  multi-joint  motions  of  the  head,  limbs,  and  trunk  in  3-D 
space  have  evolved  rapidly  over  the  last  decades,  engendering  a  paradigm  shift  in  the  field  of 
motor  control  away  from  the  study  of  simple  movements  such  as  single  joint  motions  to  the 
study  of  more  complexly  coordinated,  multi-joint  naturalistic  movements.  Positions  of  literally 
dozens  of  points  on  the  limbs  and  body  can  now  be  captured  with  high  spatial  and  temporal 
resolution.  As  well,  lightweight,  wearable  eye  tracking  systems  incorporating  tiny 
microelectronic  scene  and  eye  cameras  worn  on  an  eyeglass  frame  are  becoming  available 
(http://www.arringtonresearch.com/scene.html). 

B. 3  Addressing  the  failure  to  record  the  whole  of  physical,  mental,  and  physiological 
behavior  that  the  brain  controls  in  sufficient  detail  and  across  a  sufficient  breadth  of 
circumstances. 

The  intent  is  to  include  and  leverage  information  from  such  sources  as  overt  motor  behavior,  eye 
movement,  heart  function  (electrocardiography  [ECG]),  muscular  activity  (electromyogram 
[EMG]),  and  brain  activity  (EEG),  as  well  as  context  cues  regarding  the  multisensory  task 
environment,  in  order  to  infer  intentionality  and  examine  the  cognitive  substrates  on  which 
motivated,  goal-directed  action  is  based.  Only  by  combining  these  sources  meaningfully  can  the 
information  latent  in  the  recorded  brain/body/system  dynamics  be  identified. 

We  believe  that  the  study  of  motivated  actions,  performed  within  normal  3-D  environments  via 
combined  high-density  EEG,  full-body  motion  capture,  and  eye  tracking,  essentially  constitutes  a 
new  brain  imaging  modality,  for  which  members  of  our  Alliance  team  from  the  UCSD  (S. 
Makeig,  lead)  have  recently  proposed  the  term  “mobile  brain/body  imaging”  (MoBI)  (Makeig  et 
ah,  2009). 
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Within  the  CaN  CTA  program,  MoBI  data  acquisition  will  include,  but  not  be  limited  to,  the 
following: 

•  High- density  EEG.  EEG  is  the  only  brain  imaging  modality  with  high  temporal  and  fine 
(potentially  cm  -scale)  spatial  resolution  that  is  lightweight  enough  to  be  worn  in 
operational  settings.  Because  of  historical  limitations  in  the  size,  weight,  and  cost  of 
available  high-density  EEG  systems,  early  brain-computer  interface  (BCI)  systems  have 
been  designed  to  use  a  minimal  number  of  recording  channels.  Recent  advances  in  sensor 
and  data  acquisition  design  make  mobile  high-density  EEG  recording  viable  as  a  potential 
measurement  modality.  High-density,  as  opposed  to  low-density,  EEG  data  are  integral  to 
the  development  of  neuroergonomic  systems  as  they  provide  a  larger  number  of  degrees  of 
freedom,  allowing  much  more  flexibility  in  segregating  brain  EEG  sources,  within  the 
context  of  the  advanced  data  analytic  and  modeling  approaches  being  developed  within  the 
CTA  program. 

•  Whole -body  motion  capture  (mocap).  To  interpret  the  function  and  meaning  of  brain  EEG 
source  signals,  we  need  to  know  and  understand  what  processes  these  signals  are  actively 
supporting  or  controlling.  A  most  important  part  of  this  is  physical  behavior,  yet 
heretofore  little  brain  research  has  focused  on  the  high-density  brain  dynamics  supporting 
unrestricted  physical  activity — in  particular,  the  distributed  brain  dynamics  supporting 
motivated,  whole-body  action.  Recent  technical  developments,  however,  allow  recording 
of  robust  EEG  signals  from  electrodes  placed  on  the  head  surface  in  combination  with 
detailed  motion  capture  techniques.  Understanding  what  aspects  of  the  EEG  relates  to 
control  of  bodily  actions  allows  better  identification  of  those  aspects  of  EEG  source  signals 
that  index  cognitive  processes,  including  appraisal  or  evaluation  of  sensory  events,  and  the 
motivation  and  intent  of  physical  actions. 

•  Wearable  eye  gaze  tracking.  An  aspect  of  behavior  monitoring  that  is  most  important  for 
understanding  and  monitoring  cognitive  processes  involving  visual  perception  is  eye  gaze 
and  activity  tracking.  The  human  visual  system  processes  both  static  and  kinetic  portions 
of  the  visual  scene  through  a  series  of  typically  3/s  to  4/s  saccadic  jumps  and  fixations. 

The  brain  largely  uses  information  gained  during  fixations  to  build  and  maintain  3-D  visual 
awareness  and  direct  active  visual  exploration  of  the  environment,  although  the  ways  in 
which  it  does  this  are,  so  far,  little  understood. 

The  main  CES  experiment  to  be  conducted  at  UCSD  may  involve  an  extended  operational  game 
space  in  which  ally,  foe,  and  indifferent  agents  will  move  and  act.  All  experimental  events  will 
be  recorded  along  with  the  MoBI  data  to  create  a  large  database  for  studying  relationships 
between  operator  attention-shifting  and  decision-making  success,  and  model  neurobehavioral 
differences  between  subjects. 
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B. 4 Addressing  the  lack  of  mathematical  modeling  methods  and  software  to  find  statistical 
relationships  between  moment-to-moment  variations  in  environmental,  behavioral,  and 
functional  brain  dynamic  recordings. 

The  planned  research  approach  within  the  CaN  CTA  is  one  among  many  new  scientific 
initiatives  now  becoming  feasible  because  of  accelerating  advances  in  statistical  or  information- 
based  signal  processing  and  modeling.  Many,  if  not  all,  of  these  new  methods  can  be  derived 
from  Bayesian  models  of  information  contained  in  and  conveyed  by  multidimensional  data  under 
various  modeling  assumptions.  Statistics,  once  dominated  by  the  need  for  simplifying  (second- 
order  Gaussian)  assumptions  required  to  retain  closed-form  analytic  tractability,  can  now  use 
much  more  data-adaptive  and  computationally-intensive  statistical  approaches.  Such  approaches 
allow  automated  discovery  of  joint  higher-order  structure  in  data.  In  particular,  individual 
differences  between  subjects  in  an  experiment  no  longer  need  to  be  treated  as  a  cloud  of 
Gaussian  “noise.”  Nor  do  differences  between  experimental  conditions  need  to  be  isolated  by 
the  experimental  design  and  then  separately  tested  in  different  conditions  using  orthogonal 
planned  comparisons.  Within  the  CaN  CTA  program,  analytic  approaches  to  characterizing  and 
interpreting  high-dimensional,  MoBI  data  will  include  the  following: 

•  Information-based  signal  processing.  In  particular,  the  independent  component  analysis 
(ICA),  sparse  Bayesian  inverse,  and  new  “context  ICA”  methods  under  development  by 
UCSD  Consortium  members  can  allow  automated  discovery  of  relationships  between  brain 
dynamics  and  experimental  contexts  even  in  multimodal  data  from  complex  experiments. 
Further,  individual  subject  differences  revealed  by  the  data  can  be  subjected  to  the  same 
discovery  approach,  thereby  determining  from  the  data  themselves,  rather  than  pre¬ 
supposing,  the  types  and  directions  of  relevant  between- subjects  differences.  The  CaN 
CTA  program  will  build  on  these  new  methods,  discovering  the  behavioral  and  brain 
dynamic  patterns  exhibited  by  individual  operators  in  certain  types  of  circumstances  and 
then  building  multidimensional  pattern  recognition  algorithms  to  monitor  the  appearance  of 
these  reactions  during  operationally  relevant  test  sessions  and,  eventually,  field  operations. 

•  Identifying  Links  Between  Behavior  and  EEG  Dynamics.  Interpreting  the  multimodal 
MoBI  data  described  above  requires  development  of  adequate  methods  for  modeling 
relationships  between  rapidly  changing  high-dimensional  brain  source  activities  and  the 
complexities  of  natural  motor  behavior.  To  discover  relationships  between  high¬ 
dimensional,  synchronously  recorded  brain  and  body  movement  data,  they  first  should  be 
nonlinearly  transformed  in  ways  appropriate  to  the  nature  and  origin  of  each  type  of  data. 
Then,  the  structure  of  the  transformed  joint  data  may  be  explored  using  data-information- 
based  machine  learning  methods  (Baker  et  ah,  2005)  and  EEG  brain  sources  imaged  using 
statistical  inverse  imaging  methods  (Michel  et  ah,  2004;  Wipf  et  ah,  2007).  Modeling 
transient  coupling  between  activities  of  distributed,  quasi-independent  sources  in  different 
brain  areas  is  a  further  important  dimension  of  data  mining  and  modeling  research. 
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•  Multimodal  Brain  Imaging.  As  MoBI  technology  and  analysis  methodologies  are 
developed,  investigations  using  a  wide  range  of  experimental  paradigms  will  become 
possible,  perhaps  beginning  with  simple  motivated  actions  such  as  3-D  orienting,  pointing, 
and  grasping  (Hammon  et  ah,  2008),  and  finally  extending  to  a  wide  range  of  tasks  and 
natural  behaviors,  including  biomechanical  adaptation  and  learning,  navigation,  and  social 
interactions  in  real-world  operational  environments.  Though  many  existing  experimental 
designs  in  all  these  areas  might  be  fruitfully  exploited,  the  tight  coupling  between  EEG, 
novelty,  and  appraisal  of  events  suggests  that  these  factors  receive  more  explicit  attention 
in  MoBI  experiment  designs.  The  result  of  combining  these  sensor  development,  data 
analysis,  and  source  imaging  technologies  should  be  the  development,  in  coming  years,  of 
truly  dynamic,  higher-definition  imaging  of  distributed  EEG  brain  dynamics  supporting 
natural  cognition  and  behavior. 

•  Exploitation  of  Graph  Dynamics.  In  the  case  of  MoBI  EEG  real-time  data  capture,  we  will 
exploit  our  ability  to  extract  distributed  patterns  (graphical  constellations)  of  cortical 
dipole-patch  activity  using  our  ICA  and  sparse  Bayesian  learning  (SBE)  inverse  problem 
solver,  to  determine  appropriate  graph  invariants  (see  Ehrenfeucht,  Harju,  and  Rozenberg, 
1997;  Graph  Properties,  n.d.).  We  will  work  with  graph  invariants  (GI)  appropriate  for 
weighted  as  well  as  unweighted  graphs,  and  will  experimentally  determine  those  GIs  that 
are  maximally  robust  to  reasonably  sized  movements  of  EEG  and  body  movement  sensors 
and  changes  in  EEG  connectivity  produced  by  sweat,  grime,  etc.  A  key  intent  of  this 
approach  is  to  greatly  reduce  the  need  for  non-robust,  baseline  pre-calibration  and  tuning  of 
the  EEG  sensors.  We  will  also  determine  a  list  of  GIs  that  encode  independent  information 
highly  correlated  with  operator  cognitive  performance. 

•  Hierarchical  Bayesian  Network  Modeling.  Using  either  the  raw  time  series  of  cortical 
dipole-patch  activity  data  or  times  series  of  graph  invariants,  we  will  use  both  hierarchical 
Bayesian  networks  to  obtain  or  further  enhance  higher-level  features  associated  with 
cognitive  and  affective  state.  Such  processing  can  also  produce  features  that  are  robust  to 
sensor  and  environmental  degradation  or  be  used  to  enhance  the  stability  of  GI  features.  At 
every  level  of  the  Bayes  network,  we  will  have  a  time  series  of  abstracted  state  information 
will  be  used  to  estimate  current,  and  predict  future,  operator  state  using  the  theory  of 
Markov  decision  processes  (MDP).  Eor  any  given  human  operator,  a  MDP  model  can  be 
refined  to  predict  how  that  operator  will  respond  in  a  specific  situation.  Eor  example,  it 
should  be  possible  to  determine  if  the  operator  is  risk-preferring,  risk-neutral,  or  risk- 
adverse.  It  may  also  be  possible  to  predict  how  much  stress  an  operator  can  sustain  before 
he  experiences  a  “fight-or-flight”  reaction  and  which  of  the  two  alternatives  the  operator  is 
more  likely  to  engage  in. 
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B.  5 Addressing  the  lack  of  sufficient  data  archives  and  resources  to  systematically  study 
relationships  between  individualized  models  derived  for  cognitive  monitoring  and 
individual  differences  in  performance,  cognitive  ability,  and  personality,  relationships 
that  could  optimize  performance  of  cognitive  monitoring  systems  while  lowering 
requirements  for  collecting  individual  subject  training  data. 

An  important  source  of  uncertainty  here  is  the  extent  to  which  individualized  brain/behavior 
models  capable  of  accurately  estimating  Soldier  cognitive  state  may  require  individualized 
model  training  data  that  may  be  prohibitively  expensive  to  collect  routinely,  while  having  a 
prohibitive  cost  in  model  performance  without  such  training  data.  A  way  out  of  this  dilemma  is 
to  apply  collaborative  filtering  principles  (Han  and  Kamber,  2006)  recently  developed  to  use 
large  data  archives  (including  web  archives  and  personal  information  archives)  to  build  more 
accurate  and  complete  models  of  individuals  from  fragmentary  data  by  considering  the  totality  of 
information  available  about  other  persons  like  the  person  of  interest  (e.g.,  in  the  same  physical  or 
categorical  “neighborhood”).  The  CaN  CTA  will  collect  a  database  of  brain/behavioral  data 
from  a  sufficient  number  of  subjects  to  allow  exploration  of  the  value  of  this  approach  for 
individualized  cognitive  monitoring  in  operational  settings  without  requiring  a  prohibitive 
amount  of  personal  training  data  to  create  models  for  new  Soldier  subjects. 

Overarching  Considerations  for  Addressing  Scientific  Barriers 

Outlined  above  is  a  MoBI-based  approach  to  addressing  five  scientific  barriers  to  studying  and 
precisely  quantifying  the  natural  interactions  among  Soldier  brain,  behavioral,  sensory,  and 
performance  dynamics  as  situated  within  the  operational  environment.  The  successful 
implementation  of  such  an  approach  must  take  into  account  several  factors: 

•  EEC  sources.  Scalp  EEG  signals  sum  source  activities  arising  within  cortical  domains 
whose  local  field  activity  becomes  partially  synchronized,  giving  rise  to  far-field  potentials 
that  each  project,  by  volume  conduction,  to  nearly  all  the  scalp  electrodes,  where  they  are 
summed  with  differing  strengths  and  polarities  (Makeig  et  ah,  2004a). 

•  EEG  and  movements.  It  seems  likely  that  the  brain  may  use  naturally  emergent  local  field 
synchronies  (cortical  EEG  sources)  to  focus  its  extremely  high-dimensional  synaptic  scale 
activity  onto  much  lower-dimensional  control  of  motor  behavior.  If  so,  then  regular 
relationships  between  EEG  and  motor  actions  may  be  found  in  MoBI  data.  Particularly 
salient  relationships  between  EEG  and  body  movements  may  occur  at  movement  decision 
points,  especially  pronounced  when  movements  are  distinctly  motivated,  for  example, 
quick  reactions  to  threatening  events. 

•  Loci  of  expected  effects.  Many  cortical  regions  are  interactively  involved  in  supporting 
motivated  motor  behavior — not  only  primary  motor  areas  directly  supporting  brain  motor 
commands,  but  also  areas  supporting  motor  planning  and  expectation,  perceptual  motor 
and  sensorimotor  integration,  and  spatial  awareness  and  executive  function,  including  areas 
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directly  connected  to  sub-cortical  brain  “valuation”  systems  that  support  rapid  behavioral 
adjustments  to  anticipated  or  potential  threats,  rewards,  and  errors. 

•  Correlation  versus  causation.  Observed  relationships  between  EEG  changes  and  motor 
actions  may  not  always  reflect  a  direct  neural  coupling.  Eor  example,  continual  changes  in 
the  cortical  distribution  of  alpha  band  power  during  inquisitive  movements  may  index 
concurrent  shifts  in  the  distribution  of  sensory  attention  (Worden  et  ah,  2000)  rather  than  or 
in  addition  to  motor  planning  or  execution  processes. 

•  EEG  artifacts.  A  primary  challenge  to  performing  EEG  brain  imaging  in  mobile 
circumstances  is  extracting  meaningful  event-related  brain  dynamics  from  signals  that 
necessarily  include  significant  non-brain  artifacts  arising  from  subject  eye  movements, 
head  and  neck  EMG  activity,  cardiac  artifacts,  line  noise,  and  other  non-brain  sources.  A 
signal  processing  approach  developed  over  the  last  two  decades,  ICA  (Bell  and  Sejnowski, 
1995)  has  proven  to  be  effective  for  EEG  decomposition  into  functionally  and 
physiologically  distinct  source  activities  (Makeig  and  Jung,  1996;  Makeig  et  ah,  2002). 

ICA  decomposition  is  a  data-driven  method  that  results  in  a  set  of  spatial  filters,  each  of  which 
passes  information  from  a  distinct  information  source  in  the  recorded  multichannel  data  (Makeig 
et  ah,  2004a).  In  particular,  ICA  can  be  an  effective  method  of  identifying  and  separating  several 
classes  of  artifacts  (Jung  et  ah,  2000).  Under  favorable  circumstances,  ICA  decomposition  of 
continuous  or  discontinuous  high-density  EEG  data  also  allows  the  separate  and  concurrent 
monitoring  of  dozens  of  EEG  brain  as  well  as  non-brain  artifact  sources,  thus  avoiding  much  of 
the  often  severe  data  reduction  integral  to  traditional  methods  that  reject  from  analysis  EEG  data 
epochs  containing  movement  artifacts. 

Applied  to  data  from  mobile  participants,  ICA  can  separate  and  monitor  EMG  activity  from 
individual  head  and  neck  muscles  that,  since  they  do  not  move  appreciably  and  have  spatially 
fixed  patterns  of  projection  to  the  EEG  electrodes.  ICA  can  also  isolate  into  a  small  subspace 
other  artifacts  whose  projections  to  the  recording  array  are  not  static  but  move  in  spatially 
stereotyped  patterns,  for  example,  slow  blinks  or  cardiac  artifacts.  Spatially  labile,  non- 
stereotyped  artifacts,  however,  can  quickly  spew  hundreds  of  unique  scalp  distributions  into  the 
data,  each  de  facto  independent  of  the  rest  of  the  recorded  data  and  not  separable  into  a  low¬ 
dimensional  independent  component  (IC)  subspace.  Eor  example,  such  artifacts  may  result  if 
extreme  head  and  scalp  movements  produce  small  movements  of  many  of  the  electrodes  on  the 
scalp.  Such  spatially  non-stereotyped  artifacts  need  to  be  identified  and  removed  from  the 
training  data  before  or  during  ICA  decomposition  (Onton  et  ah,  2006). 

•  Brain  data  preprocessing.  Scalp-recorded  EEG  signals  are  each  mixtures  of  activity  from 
a  variety  of  brain  as  well  as  non-brain  sources,  and  the  number  of  possible  brain  source 
domains  (e.g.,  cortical  patches)  is  quite  large.  Thus,  the  problem  of  identifying  the 
unknown  EEG  source  signals  and  their  individual  projections  to  the  scalp  sensors  from  the 
data  is  a  difficult  blind  source  separation  and  physical  inverse  problem.  Methods  and 
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software  for  imaging  the  source  dynamics  of  cortical  activity  from  high-density  scalp 
recordings  are  steadily  evolving  (Michel  et  ah,  2004).  Applied  to  EEG  data,  ICA 
algorithms  result  in  a  set  of  spatial  filters  that  linearly  separate  EEG  into  a  sum  of 
component  processes  with  maximally  temporally  independent  time  courses  (Makeig  et  ah, 
1996;  Makeig  et  ah,  2004a;  Makeig  et  ah,  2004b). 

Many  IC  source  processes  project  to  the  scalp  with  a  nearly  “dipolar”  pattern  compatible  with  a 
cortical  patch  source.  Spatial  equivalent  dipole  location  (in  or  near  brain,  in  or  near  eyes,  or 
scalp  muscles)  can  be  estimated  using  an  electrical  head  model,  optimally  one  built  from  the 
participant  MR  head  image  (Mosher  et  ah,  1999).  Open  source  software  is  available  (Delorme 
and  Makeig,  2004),  and  use  of  anatomic  magnetic  resonance  (MR)  subject  head  images  can 
produce  the  most  accurate  source  location  estimates  (Akalin,  Acar  and  Makeig,  2010). 

However,  from  the  point  of  view  of  dynamic  analysis,  ICA  decomposition  into  separate  source 
activities  can  also  be  considered  a  signal  pre-processing  step  to  enable  further  joint  analysis  of 
the  brain  EEG  and  behavioral  data. 

•  Movement  data  preprocessing.  Pre-processing  of  body  motion  capture  data  is  likewise 
non-trivial.  Modern  biomechanical  data  analysis  proceeds  from  recording  the  changing 
positions,  velocities,  and/or  accelerations,  in  external  world  or  body-centered  coordinates, 
of  sensors  placed  on  the  body  surface,  to  computing  movements  of  each  body  and  limb 
segment  relative  to  another  in  a  body-centered  reference  frame,  to  estimating  the  time 
courses  of  the  particular  muscular  forces  that  produce  those  joint  movements  (Poizner  et 
ah,  1995;  Soechting  and  Elanders,  1995).  Determining  joint  movements  from  motion 
capture  records  is  a  mathematical  inverse  problem,  as  a  kinematic  transformation  is  needed 
between  trajectories  of  the  recorded  body  surface  positions  and  those  of  the  underlying 
joint  angles  (Soechting  and  Elanders,  1992).  Determining  the  muscular  torques  applied  to 
the  limb  segments  to  produce  the  observed  joint/limb  trajectories  is  a  further  inverse 
problem  requiring  a  biomechanical  model  of  the  body  skeleton  and  musculature  (Winter 
and  Eng,  1995),  for  which  open  source  software  is  becoming  available  (Delp  et  ah,  2007). 

It  is  not  clear,  however,  the  extent  to  which  EEG  changes  relate  more  directly  to  changes  in 
body  or  body-part  position,  movement,  or  muscular  activation — or  to  some  combination  of 
these.  This  uncertainty  complicates  the  development  of  mobile  brain/body  imaging  models 
of  human  action. 

3.2  Translational  Science  Within  Specific  Neurocognitive  Performance  Domains 

As  stated  before,  our  research  program  targets  operations  in  complex,  mixed-initiative  system 
environments  in  which  dynamic,  complex  information  streams  must  be  integrated  across 
multiple  spatial  and  temporal  scales.  To  address  this  target,  the  research  much  address  three 
main  objectives  as  preconditions  on  solving  these  problems:  help  the  Soldier-system  (1)  actively 
maintain  situational  awareness  of  the  immediate  and  larger-scale  operational  environment  based 
on  ever-changing  information  delivered  concurrently  via  multiple  sensory  channels  (i.e.. 
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visual, auditory,  tactile);  (2)  make  sound  decisions  and  develop  appropriate  courses  of  action  in 
response  to  unfolding  events  in  their  environment,  through  improved  understanding  of  sensory 
and  cognitive  processing  achieved  by  combining  real-time  analysis  of  operator  brain  activity, 
body  and  eye  movements,  and  behavior,  with  information  gleaned  from  analysis  of  the  sensory 
streams  the  operators  are  monitoring;  and  (3)  take  into  account  inter-  and  intra-individual 
variability  in  the  neurocognitive  and  behavioral  strengths  and  strategies  of  operators  to  optimize 
Soldier-system  information  exchange  to  improve  the  likelihood  of  success  associated  with 
making  risky  decisions. 

While  the  previous  section  outlined  a  MoBI-based  approach  to  overcoming  the  barriers  to 
understanding  brain-behavior-environment  interdependencies,  this  section  outlines  the 
neurocognitive  performance  domains  in  which  the  CaN  CTA  research  focuses  on  in  order  to 
accomplish  our  main  objectives  and  translate  basic  neuroscience  to  militarily  relevant 
environments.  The  performance  domains  include  the  following: 

•  Alertness.  In  low  or  constant  information  settings,  frank  drowsiness  and  even  sleep  onset 
may  accompany  intentions  to  remain  alert  and  attentive.  Jung  et  al.  (1997)  showed  that 
human  EEG  contains  abundant  and  stable  information  sufficient  to  alert  a  computerized 
system  that  its  human  operator  is  drowsy  or  somnolent — information  that  is  still  not 
integrated  into  human-system  interface  designs — though  in  a  wide  range  of  settings, 
contra-circadian  demands  on  operator  time  and  attention  continue  to  increase,  with 
statistically  probable  and  likely  under- appreciated  consequences  for  overall  human-system 
performance  including  increased  risk  of  disasters  produced  by  human  performance  failure 
(“man  out  of  the  loop”). 

•  Attention.  We  are  likely  little  aware  of  the  extent  to  which  we  are  continually 
redistributing  our  attention  among  sensory,  mnemonic,  and  cognitive  channels.  Eailure  to 
properly  scan  the  environment  for  crucial  information  is  a  hallmark  of  performance  under 
stress.  Even  physically  scanning  across  visual  information  sources  with  our  eyes  does  not, 
however,  guarantee  that  we  effectively  attend  to  significant  visual  information  within  our 
field  of  view.  Eor  example,  we  may  be  at  the  time  primarily  focusing  on  receiving  and 
evaluating  audio  information,  or  we  may  be  daydreaming  about  a  pleasant  memory  or 
anticipated  event.  More  research  is  needed  to  determine  to  what  extent  the  depth, 
distribution,  and  dynamics  of  attention  may  be  modeled  and  accurately  estimated  from 
MoBI  data.  A  phenomenon  to  exploit  is  the  known  appearance  of  alpha  band  (8-12  Hz) 
EEG  activity  in  cortical  regions  used  to  monitor  relatively  unattended  sensory  channels 
(Gomez-Ramirez  et  al.,  2011;  Kelly  et  ah,  2006). 

•  Arousal.  The  relationship  between  performance  and  arousal  has  long  been  known  to 
assume  a  well-known  inverted  U  shape,  with  both  under-aroused  and  over-aroused  interest 
and  attentiveness  dissociated  from  good  performance.  Under-arousal  leads  to  failure  to 
explore  the  sensory  environment  and  recognize  consequences  of  stimulus  or  event 
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combinations,  while  over-arousal  leads  to  “tunnel  vision”  fixation  on  a  particular  fact  or 
input  or  even  an  inability  to  focus  on  external  inputs  at  all.  Likely  phenomenon  to  exploit 
here  is  the  appearance  of  coherent  beta  band  (14-28  Hz)  LEG  activities  in  sensory  cortex 
during  aroused  attention  (Bekisz  and  Wrobel,  1993). 

•  Appraisal.  Understanding  the  cognitive  significance  of  operator  EEG,  eye  movement,  and 
other  behavioral  processes  logically  requires  knowledge  and  understanding  of  the  sensory 
experience  of  the  operator.  This  is  perhaps  the  most  challenging  of  the  inverse  problems 
involved  in  the  development  of  neuroergonomic  systems,  as  robust  computer  interpretation 
of  the  significance  of  complex  natural  sensory  information  (visual,  auditory,  etc.)  is  not  yet 
within  reach.  However,  interpretation  of  EEG  signals  recorded  during  shifting  of  attention 
between  any  combination  of  sensory  channels  may  be  aided  by  even  partial  information, 
e.g.,  about  the  degree  of  predictability  of  stimulus  events  that  may  be  gleaned  from  the 
sensory  stimuli  themselves. 

Recent  work  in  computational  linguistics  has  shown  that  statistically  defined  linguistic  surprisal 
(the  inverse  of  sequential  word  probability)  can  determine  the  length  of  individual  fixations 
during  text  reading,  and  direct  relations  between  linguistic  surprisal  and  EEG  are  now  well 
established.  In  addition,  automated  speech  to  text  translation  is  now  relatively  advanced,  as  is 
automatic  grammatical  parsing  from  text.  More  basic  research  is  needed,  however,  in  EEG 
dynamics  accompanying  speech  listening  and  text  reading.  Understanding  which  aspects  of  the 
EEG  signal  are  related  to  what  is  heard  (or  read)  will  help  to  isolate  those  signal  aspects  that 
index  an  operator’s  (“top-down”)  appraisal  of  the  speech  content. 

Recent  work  in  machine  vision  has  shown  the  possibility  of  defining  screen  regions  of  bottom-up 
visual  interest  in  video  images  via  statistical  definitions  of  visual  surprise,  and  these  have  been 
shown  to  well  predict  locations  of  eye  fixation.  Basic  research  is  needed  to  better  understand 
which  aspects  of  EEG,  psychophysiological,  and  behavioral  signals  relate  to  top-down  visual 
search  versus  responses  to  events  of  bottom-up  visual  interest,  and  interpret  visual  information 
acquired  during  individual  eye  fixations. 

A  potentially  valuable  capability  for  MoBI-enabled  human  system  interface  designs  is  the 
capacity  to  detect  self-perceived  or  unperceived  failures  to  appraise  the  significance  of  sensory 
events  or  event  combinations.  A  simple  case  might  be  failure  of  brain  activity  or  body 
movements  to  register  an  alerting  reaction  to  delivery  of  a  signal  to  the  operator  known  to  require 
urgent  evaluation  (e.g.,  a  “red  warning  light”).  A  more  complex  example  would  be  the  capacity 
to  detect  confusion  in  an  operator  presented  with  confusing,  incomplete,  and/or  contradictory 
information. 

•  Affect.  Recent  EEG  analysis  has  begun  to  show  that  our  emotional  or  affectively  laden 
brain  responses  to  events  that  our  brain’s  limbic  system  immediately  characterizes  as 
“good”  or  potentially  “bad”  news  can  be  discerned  from  high-density  EEG  data  (Onton  and 
Makeig,  2009).  We  are  quite  sensitive  to  teammates’  affective  characterization  of  events, 
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as  well — a  co-worker’s  sudden  intake  of  breath  may  immediately  arouse  us  to  shift  our 
attention  to  find  the  cause  of  their  affectively  laden  response.  Recent  demonstrations  have 
shown  that  automated  recognition  of  unexpected  negative  events  (for  example,  sudden 
failures  of  a  computer  game  program  to  follow  operator  instructions)  can  be  automatically 
detected  from  scalp  EEG  signals  and  may  be  used  in  a  human-computer  interface  to 
improve  overall  human-system  performance.  Rapid  fluctuations  in  brain  “good/bad” 
responses  to  events,  if  made  known  to  a  computer  information  system  an  operator  is 
controlling,  could  be  used  to  selectively  change  display  patterns,  for  example,  while  the 
absence  of  an  affectively  laden  brain  response  to  an  event  the  system  knows  to  be 
important  (e.g.,  “Warning  light  on!”)  could  cue  further  useful  system  actions.  Detection 
and  representation  of  the  unexpected  abnormal  mood  of  an  out-of-sight  team  member 
might  foster  better  team  cohesiveness  and  performance.  Brain  affective  responses  of 
expert  observers  (including  mental  “Oh  no!  ’s”)  might  also  be  used  to  give  more  effective, 
near-instant  and  continuous  feedback  to  robotic  systems  that  are  being  trained  to  perform 
through  experience  than  manual  expert  feedback  delivery  methods. 

•  Agency.  Recent  work  has  shown  that  intention  to  make  a  voluntary  movement  to  the  left  or 
right  produces  distinct  and  recognizable  patterns  of  electrical  activity  on  the  parietal  scalp, 
consistent  with  locations  of  spatial  planning  areas  found  in  invasive  animal  and  human 
fMRI  brain  studies  (Wang  and  Makeig,  2009).  An  information  display  system  that  can 
anticipate  the  motivated  actions  of  its  operator(s)  might  be  designed  to  be  easier  and  more 
efficient  to  operate  than  a  passive  system  with  no  knowledge  of  operator  intent. 

Cognitive  and  affective  aspects  of  operator  intent  in  making  an  action  may  also  be  crucial  in 
optimizing  system  response.  Eor  example,  might  a  system  detect  the  fact  that  an  operator 
perceives  he/she  has  pressed  a  button  in  error,  then  give  the  operator  an  opportunity  to  confirm 
or  rescind  the  command?  Else,  by  interpreting  the  speed  of  a  motor  selection  gesture  plus 
accompanying  brain  dynamics  consistent  with  perceived  urgency,  might  a  system  be  able  to 
usefully  refocus  its  information  delivery  capabilities  to  maximize  delivery  of  relevant 
information  and  feedback,  as  a  human  colleague  might  do? 


4.  Transitions 


Over  the  course  of  the  CaN  CTA,  the  Alliance  will  perform  translational  research  that  builds  on 
the  information  gained  through  basic  research  and  is  envisioned  to  result  in  practical  solutions 
that  address  particular  Army  problems  or  unmet  needs.  Potential  endpoints  of  the  envisioned 
research  are  both  products  that  ultimately  are  integrated  into  Army  systems  and  new  pathways  to 
discovery,  which  would  include  new  methods  of  investigation;  new  approaches  for  summarizing, 
visualizing,  and  analyzing  data;  new  measures  or  biological  markers  for  detecting  specific 
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cognitive  states  or  processes  such  as  fatigue,  attention  allocation,  working  memory,  stress, 
learning,  etc.;  and  new  algorithms  for  differentiating  among  such  states. 

This  section  outlines  three  planned  transition  efforts  (TEs)  to  which  the  research  projects  can 
contribute.  These  TEs  can  thus  serve  as  guideposts  for  investigators  as  they  plan  and  execute 
respective  projects.  The  three  projects  identified  were  designed  to  be  somewhat  progressive  in 
nature.  That  is,  as  a  close  examination  of  the  included  timelines  will  reveal,  the 
accomplishments  of  TE  I  will  feed  the  objectives  and  goals  underlying  the  advancement  of  TE  II 
and  so  on. 

Described  below  are  the  three  TEs,  followed  by  descriptions  of  their  background,  potential 
barriers,  timelines,  and  specific  transitions.  Additionally,  a  corresponding  Translational  Path 
section  is  provided  within  the  description  of  each  individual  research  project  in  sections  4,  5,  and 
6  that  explains  the  connection  between  each  project  and  these  TEs.  The  three  initial  planned, 
progressive  TEs  for  the  basic  research  conducted  under  the  CaN  CTA  are  as  follows: 

TE  I.  MoBI  methods  for  detecting  changes  in  vehicle  operator  alertness  and  consequent 
performance-failure  prediction  (near-term  transition  to  Army  systems) 

TE  II.  Advanced  tools  for  MoBI-based  neurocognitive  performance  assessment  (Integrated 

Army  and  Research  transition;  near-  and  mid-term  transition  to  research  tools;  mid-  and 
far-term  transition  to  Army  systems) 

TE  III.  Enhanced  design  of  high-information  multichannel  crew  stations  (far-term  Army 
transition,  5  years) 

Additional  potential  transitions,  outside  of  these  pre-defined  TEs,  will  be  identified  as 
appropriate  (e.g.,  opportunity-driven  transitions  will  be  developed  and  described  over  the  course 
of  the  CTA). 

4.1  TE  I:  MoBI  for  Detecting  Changes  in  Vehicle  Operator  Alertness  and  Performance 
Prediction 

A  consistent  and  critical  problem  for  Soldiers  lays  in  maintaining  alertness  and  vigilance  while 
performing  attention-demanding,  long-duration  tasks  such  as  driving  within  dynamic,  threat¬ 
laden  environments.  This  near-term  TE  integrates  basic  science  and  technology  into  a 
mobile/wireless  cognitive-state  monitoring  system  that  can  assess  fluctuations  in  individuals’ 
alertness  and  capacity  for  driving  performance  via  low-density  EEC  signals.  We  aim  to  integrate 
and  demonstrate  such  a  monitor  into  a  military  vehicle  test-bed. 

4.1.1  TE  I.  Background 

Since  1989,  Alliance  team  members  at  UCSD  have  conducted  fundamental  studies  (Makeig  and 
Mow,  1993;  Makeig  and  Jung,  1995,  1996;  Jung  et  ah,  1997;  Makeig  et  ah,  2000)  demonstrating 
the  feasibility  of  accurately  estimating  shifts  in  an  individual  operator’s  level  of  alertness,  as 
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indexed  by  changes  in  their  level  of  performance  on  a  simple  auditory  target  detection  task,  by 
monitoring  the  changes  in  their  EEG  power  spectra  or  other  measures.  Recently,  NCTU 
investigator  Ein  and  UCSD  investigator  Jung  have  extended  this  research  into  EEG-based 
alertness  assessment  in  a  continuous  driving  task  (Ein  et  ah,  2005,  2006). 

Alliance  team  members  have  also  conducted  basic  research  underlying  cutting-edge  technologies 
for  the  real-time  joint  recording  and  analysis  of  environmental,  behavioral,  and  functional  brain 
dynamic  variables.  Eor  example,  team  members  at  NCTU  and  UCSD  have  recently 
demonstrated  the  feasibility  of  using  dry  MEMS  EEG  sensors  and  miniaturized  supporting 
hardware/software  to  continuously  collect  EEG  data  from  four  frontal  non-hairy  scalp  sites  in  a 
realistic  VR-based  driving  simulator  (Ein  et  al.  2008). 

This  TE  will  integrate  results  from  over  a  decade  of  research  in  Alliance  partner  laboratories 
related  to  the  Neurocognitive  Performance  TA  and  the  Neurotechnology  TA,  providing  a  real¬ 
time  Soldier  monitoring  system  that  can  be  implemented  into  a  military  vehicle  test-bed.  Eurther, 
this  transition  effort  will  provide  a  near-term  demonstration  of  the  translational  process  that  can 
serve  as  a  guide  for  the  successful  transition  of  future  CaN  CTA  research  products. 

4.1.2  TE  I.  Transition  Barriers  and  Hurdles 

Previous  research  has  suggested  that  an  operator  alertness/performance-failure  prediction 
monitor  for  in-vehicle  driving  may  be  sufficiently  robust  to  be  accomplished  without  fully 
addressing  scientific  barriers  B.3-5,  though  we  expect  their  reliability  to  further  increase  as 
barriers  B.3-5  are  addressed.  This  is  one  of  the  primary  reasons  that  this  effort  was  selected  as 
the  first  TE,  and  as  a  near-term  demonstration  of  the  translational  process.  The  primary 
scientific  barriers  that  this  effort  will  address  are  as  follows. 

B.lThe  restriction  of  experimental  designs  to  highly  controlled  and  impoverished 
stimulus/response  paradigms  and  environments. 

The  monitor  must  be  robust  against  a  variety  of  subject  states,  task  conditions,  and 
environmental  scenarios  that  are  expected  in  real-world  situations. 

B.lThe  lack  of  portable,  user-acceptable  (e.g.,  comfortably  wearable),  and  robust  systems 
for  monitoring  brain  and  body  dynamics. 

The  monitor  needs  to  able  to  use  reasonably  low-density  EEG  data  robustly  and  continuously 
without  the  use  of  conductive  gels  or  skin  preparation  from  harnessed,  seated  operators  that  are 
also  free  to  dismount  their  vehicles.  Available  dry  EEG  sensor  density  is  expected  to  increase 
rapidly  in  coming  years,  and  the  value  of  spatial  filtering  possibilities  afforded  by  higher-density 
recording,  for  accuracy  in  both  non-brain  artifact  rejection  and  relevant  brain  source  monitoring, 
has  been  well  established  (e.g.,  Makeig  et  al.,  2004a).  However,  previous  results  (e.g.,  Jung  et 
ah,  1997)  suggest  that  individualized  or  even  common  low-density  EEG-based  models  can 
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provide  accurate  continuous  alertness  estimates,  at  least  in  laboratory  settings  including  use  of 
simulated  vehicle  movements  (Lin  et  ah,  2008). 

B.SThe  lack  of  sufficient  data  archives  and  resources  to  systematically  study  relationships 
between  individualized  models  derived  for  cognitive  monitoring  and  individual 
differences  in  performance,  cognitive  ability,  and  personality,  relationships  that  could 
optimize  performance  of  cognitive  monitoring  systems  while  lowering  requirements  for 
collecting  individual  subject  training  data. 

The  monitor  should  be  able  to  use  a  model  of  brain/behavioral  changes  in  alertness  not  requiring 
extensive  pilot  training  data  from  each  Soldier,  while  respecting  operator  differences  in,  e.g., 
baseline  LEG  that  may  be  present  in  recordings  from  each  new  Soldier  subject.  Applied  studies 
under  TE  I  using  a  number  of  subjects  will  allow  early  investigation  within  the  CTA  of  the 
extent  to  which  individualized  data  models  perform  better  than  a  common  data  model,  while 
computational  approaches  to  characterizing  relationships  between  individual  and  group  data, 
developed  under  B.5  will  allow  first  testing  of  methods  to  quickly  individualize  a  common 
model  from  partial  (baseline)  EEG  data  on  each  individual. 

In  addition  to  the  scientific  barriers,  the  translational  process  will  have  to  address  technical 
challenges.  This  primary  technical  challenge  that  the  operator  alertness/performance  prediction 
monitor  will  face  is  as  follows: 

•  The  monitor  must  be  compatible  with  integration  into  current  Army  equipment  in  a  manner 
that  is  physically  robust,  safe,  and  acceptable  to  the  user. 

4.1.3  TE  I.  Translational  Timeline 

The  areas  underlying  this  transition  are  threefold.  Eirst,  we  will  develop  and  test  a  wearable, 
wireless,  dry-electrode,  low-density,  EEG-based  system  that  can  be  used  in  field-able  scenarios. 
This  system  will  be  capable  of  data  acquisition,  online  artifact  cancellation  and  signal 
processing.  Second,  we  will  develop  and  validate  robust  EEG  algorithms  for  monitoring 
alertness  and  cognitive  performance  estimation  in  militarily  relevant  environments.  Third,  we 
will  integrate  a  ruggedized,  wearable,  EEG-based  system  into  a  helmet  suitable  for 
experimentation  in  a  military  vehicle  test-bed.  The  planned  transition  of  this  ruggedized, 
wearable  EEG-based  monitor  will  be  to  an  Army  applied  research  and  development  program  that 
will  implement  this  monitor  into  adaptive  brain-machine  interfaces.  This  timeline  is  depicted  in 
figure  3. 
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Figure  3.  Translational  timeline  depicting  expected  progress  in  TE  I. 

In  Year  1,  the  Alliance  translational  research  projects  for  this  TE  will  support  the  development 
and  evaluation  of  a  new  16-channel  mobile  and  wireless  EEG  system  to  acquire  signals  from  all 
parts  of  the  scalp.  In  year  2,  the  Alliance  will  (1)  test  the  long-term  signal  quality  of  EEG  signals 
in  tasks  involving  a  wide  range  of  events  requiring  active  evaluation  by  operators  in  the  real- 
world  environments,  and  (2)  develop,  test,  and  demonstrate  a  wearable/wireless  alertness 
monitoring  and  management  system.  These  efforts  are  described  in  the  Alliance  research  projects 
supporting  this  transition,  which  include  the  following: 

•  MoBI-based  Eeedback  for  Training  Automated  Systems  and  System  Operators  (MBE) 

•  Operator  Dynamics  of  Event  Appraisal  (ODE) 

•  Using  Neural  Activity  to  Predict  Attentional  Eapses  During  Multisensory  Task  Scenarios 
(PAE) 

•  Effects  of  Vehicle  Motion  and  Cognitive  Eatigue  (VME) 

•  Wearable  EEG  Development  and  Testing  (WDT) 

In  addition  to  the  research  projects,  this  transition  will  require  specific  efforts  to  address  the 
following  technical  challenges: 
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•  Through  years  1  and  2  ARL  will  lead  an  effort  with  UCSD  to  integrate  Alliance  developed 
sensors  within  a  standard  Army  helmet  for  testing  of  feasibility  and  potential  safety 
concerns.  This  effort  (lead,  Dave  Kuhn)  will  include  examining  the  feasibility  of 
integrating  several  sensor  and  data  acquisition  options  and  finally  constructing  a  prototype 
in-helmet  system. 

Critical  Points 

There  are  two  critical  points  for  the  successful  completion  of  this  project. 

C.l.  The  specifics  of  the  wearable  EEG-sensor  system  will  be  agreed  upon. 

C.2.  The  monitor  will  be  transitioned  to  the  applied  Army  research  program. 

4.1.4  TE  I.  Transitions 

This  TE  will  culminate  with  the  integration  and  demonstration  of  a  mobile/wireless  cognitive- 
state  monitoring  system  that  can  assess  fluctuations  in  individuals’  alertness  and  capacity  for 
driving  performance  via  low-density  EEG  signals  in  a  military  vehicle  test-bed.  Through  a  joint 
U.S.  Army  Tank- Automotive  Research  Development  and  Engineering  Center  (TARDEC)-ARE 
mission  program,  the  alertness/performance  monitoring  system  will  be  integrated  into  three  types 
of  applications:  (1)  an  individual -based  fatigue  mitigation  system,  (2)  a  team-based  warning 
system,  and  (3)  an  adaptive  training  system.  The  mitigation  and  warning  systems  will  be 
designed  to  enhance  Soldier-system  performance  for  driving  tasks.  The  training  system  will  be 
designed  to  reduce  training  time. 

The  longer-term  significance  of  the  TE  includes  (1)  demonstrating  a  viable  prototype  in-helmet 
system  for  high  quality  EEG  capture  and  evaluation  in  a  wide  range  of  military  relevant 
environments;  (2)  enabling  safe  and  secure  EEG-based  monitoring  in  operational  tests  to  include 
use  in  system  development,  test  and  evaluation,  and  (potentially)  in  the  field;  and  (3)  providing  a 
proof  of  principle  for  other  types  of  cognitive- state  monitoring  and  alternative  uses  such  as  in  a 
traumatic  brain  injury  (TBI)  detection  or  monitoring  system.  In  addition,  by  enabling  recording 
of  brain  dynamics  in  more  complex  operational  environments  than  is  currently  possible,  there 
will  be  broader  implications  for  later  transitions  in  other  areas  for  all  of  the  research  supporting 
the  near-term  development  of  a  mobile/wireless  cognitive-state  monitoring  system  that  can 
assess  fluctuations  in  individuals’  alertness. 

Eor  example,  having  a  capability  to  safely  and  reliably  record  brain  dynamics  of  Soldiers  in 
operational  environments  will  enable  future  investigations  of  Soldier  neurocognitive  function  in 
more  diverse  operational  environments  (relating  to  TA  I).  In  like  fashion,  transitions  from  other 
TEs  will  contribute  to  further  enhance  the  alertness  technology.  Eor  example,  developing  the 
capability  to  integrate  and  synchronize  multiple  streams  of  data  will  enable  integration  of 
additional  contextual  data  that  could  improve  the  accuracy  of  estimating  operator  alertness  just 
as  the  further  development  of  data  analysis  and  mining  tools  and  techniques  (TA  II)  may  lead  to 


31 


other  applications  for  brain  monitoring  beyond  just  fatigue  (e.g.,  detecting  shifts  in  attention, 
cognitive  load,  and  TBI). 

4.2  TE  II.  Advanced  Tools  for  MoBI  and  Neurocognitive  Performance  Assessment 

A  critical  need  for  translation  of  neuroscience  to  operational  environments  is  the  appropriate 
hardware  and  software  tools  for  collecting,  analyzing,  interpreting,  and  using  brain  imaging  data 
within  real-world  environments.  This  planned  TE  provides  tools  that  will  enable  more  advanced 
monitoring  of  Soldier  performance  in  operational  environments  as  the  research  evolves  from 
laboratory  to  simulation  to  real-world  environments.  Specifically,  this  transition  effort  focuses 
on  the  development  of  (1)  sensor  technologies  and  mobile  recording  systems;  (2)  large-scale 
databases;  (3)  hardware  and  software  for  integrating  and  synchronizing  multiple  data  streams  and 
data  formats;  and  (4)  algorithms  for  mining,  analyzing  and  visualizing  data  for  investigating 
relations  among  multivariate  data  sets  comprised  of  data  spanning  different  temporal  and  spatial 
scales  and  formats.  Two  primary  types  of  transitions  will  occur  from  this  effort:  first,  in  the  near 
and  intermediate  term,  user-friendly  research  tools  will  be  implemented  that  are  necessary  for  the 
Department  of  Defense  (DoD)  translational  neuroscience  community  and  the  scientific 
community  at  large;  and  second,  in  the  intermediate  to  long-term,  real-time  tools  will  be 
implemented  that  enable  advanced  Soldier  monitoring  and  brain-in-the-loop  system  design. 

4.2.1  TE  II.  Background 

Alliance  partners  at,  UCSD,  NCTU,  UTSA,  DCS  Corporation,  and  ARE  have  a  long  history  of 
developing  hardware  and  software  tools  for  collection,  visualization,  analysis,  and  interpretation 
of  data  ranging  from  basic  neuroscience  to  Soldier-system  performance.  Eor  example,  partners 
at  UCSD  have  a  13-year  history  in  developing  an  interactive,  freely  available  Matlab  toolbox  for 
processing  continuous  and  event-related  electrophysiological  data  (Delorme  and  Makeig,  2004; 
Makeig,  Debener,  Onton  and  Delorme,  2004;  Swartz  Center  for  Computational  Neuroscience, 
n.d.).  In  a  long-term  collaborative  effort,  partners  at  UCSD  and  NCTU  have  developed  tools  for 
data  collection,  analysis,  and  interpretation  of  EEC  data  in  a  limited-range  of  real-world  settings. 
Similarly,  an  Alliance  partner  at  UTSA  has  developed  tools  for  visualization,  analysis,  and 
management  of  multimedia  datasets  in  three  major  application  areas:  biological  applications  (in 
neuroscience,  immunology,  and  bioinformatics),  pattern  forming  systems,  and  geophysical 
systems  (Robbins,  Robinson  and  Senseman,  2004;  Robbins,  Grinshpan,  Allen  and  Senseman, 
2004;  Robbins  and  Senseman,  2003;  UTSA  Visualization  and  Modeling  Eaboratory,  n.d.).  In 
another  long-term  collaboration,  partners  at  DCS  Corp  and  ARE  have  developed  tools  for  the 
modeling  and  simulation  (Johnson,  Manteuffel,  Brewster  and  Tierney,  2007)  as  well  as  the 
collection  and  analysis  (Metcalfe  et  al,  2010)  of  data  regarding  human-system  performance  in 
militarily  relevant  tasks  and  contexts.  Beyond  software  development,  in  efforts  that  are 
becoming  integrated  with  neuroergonomics  research  (Oie  and  Paul,  2009),  ARE  and  DCS  have 
long  worked  collaboratively  with  the  U.S.  Army  TARDEC  to  develop  paradigms  and  techniques 
for  assessing  performance  in  real-world  tasks  that  use  high-fidelity  vehicle  simulations  using  a 
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6-degrees  of  freedom  (DOF)  motion  simulator  (McDowell,  Rider,  Truong,  and  Paul,  2005; 
Ruffner  et  al.,  2005).  This  TE  will  leverage  the  wide-ranging  expertise  of  Alliance  partners  to 
integrate  research  from  the  Neurocognitive  Performance  TA,  the  Advanced  Computational 
Methods  TA,  and  the  Neurotechnology  TA  research  strands  to  provide  advanced  tools  that  will 
serve  as  critical  capabilities  for  applied  researchers  and  as  foundational  capabilities  upon  which 
future  technologies  will  build. 

4.2.2  TE  II.  Transition  Barriers  and  Hurdles 

While  a  large  variety  of  advanced  imaging  and  computational  tools  and  techniques  have  been 
developed  and  successfully  employed  within  the  neuroscientific  community,  the  vast  majority  of 
the  technologies  used  have  been  limited  to  laboratory,  fixed-environment  contexts  for  the 
purpose  of  offline  processing  and  analysis  of  neurobehavioral  function.  Thus,  further  work  is 
needed  towards  developing  analysis  approaches  that  provide  insight  into  neurobehavioral 
function  as  humans  continuously  interact  with  dynamic,  high-information  content  environments. 
There  has  been  little  appreciation  of  the  problems  encountered  when  assessing  neurocognitive 
performance  in  real-time  within  real,  dynamic  environments.  Neither  has  much  attention  been 
given  to  the  particular  challenges  associated  with  the  real-time  application  of  multiple  sensor 
technologies  for  joint  analysis  of  several  typically  high-density  sources  of  information  regarding 
brain  and  behavioral  dynamics.  This  tool  development  TE,  therefore,  will  have  to  address  all 
five  primary  scientific  barriers. 

B.lThe  restriction  of  experimental  designs  to  highly  controlled  and  impoverished 
stimulus/response  paradigms  and  environments. 

Tools  must  advance  beyond  those  developed  for  highly  controlled  and  impoverished  scenarios. 
Tools  developed  under  traditional  paradigms  may  not  be  robust  enough  for  application  in  highly 
complex  environments  or  multi-tasking  scenarios,  or  for  operation  in  multiple  or  fluctuating 
performance  domains. 

B.lThe  lack  of  portable,  user-acceptable  (e.g.,  comfortably  wearable),  and  robust  systems 
for  monitoring  brain  and  body  dynamics. 

Multi-purpose  acquisition  systems  need  to  able  to  collect  high-fidelity  EEG  data  robustly  and 
continuously  without  the  use  of  conductive  gels  or  skin  preparation.  Eurther,  portable  systems 
must  robustly  handle  electrical  and  motion  artifacts  that  pose  threats  to  reliable  and  valid 
recording  of  brain  dynamics  in  operational  environments. 

B.  3  The  failure  to  record  the  whole  of  physical,  mental,  and  physiological  behavior  that  the 
brain  controls,  and  the  physical  and  socio -cultural  effects  of  the  environment  that  impact 
brain  function,  in  sufficient  detail  and  across  a  sufficient  breadth  of  circumstances. 

Data  acquisition  systems  must  be  generic  or  adaptable  enough  to  collect  data  in  the  sufficient 
detail  and  across  a  breadth  of  circumstances.  Such  systems  must  incorporate  a  wide-range  of 


33 


sensor  technologies  and  thus  must  address  issues  associated  with  sampling  rate,  synchronization, 
data  storage,  and  subsequent  data  retrieval.  Specifically,  multi-aspect  datasets  collecting  high- 
density  data  from  differing  types  of  physiological  sources  (e.g.,  EEG,  EMG,  eye  and  body 
movements)  are  often  very  large  and  unwieldy  in  raw  form. 

B.4The  lack  of  mathematical  modeling  methods  and  software  to  find  statistical  relationships 
between  moment-to-moment  variations  in  environmental,  behavioral,  and  functional 
brain  dynamic  recordings. 

Due  to  the  novelty  and  the  scale  of  the  large,  multi-aspect  datasets  expected  from  within 
operationally  relevant  scenarios,  new  tools  for  data-reduction,  mining,  and  visualization  must  be 
developed.  In  particular,  as  systems  progress  towards  real-time  implementations,  such  tools 
must  enable  developers  to  constrain  system  designs  to  utilize  only  the  necessary  and  sufficient 
data  as  opposed  to  online  mining  and  processing  of  overwhelming  amounts  of  data. 

B.SThe  lack  of  sufficient  data  archives  and  resources  to  systematically  study  relationships 
between  individualized  models  derived  for  cognitive  monitoring  and  individual 
differences  in  performance,  cognitive  ability,  and  personality,  relationships  that  could 
optimize  performance  of  cognitive  monitoring  systems  while  lowering  requirements  for 
collecting  individual  subject  training  data. 

The  tools  designed  and  tested  under  TE II  should  enable  the  use  of  models  of  brain/behavioral 
changes  not  requiring  extensive  pilot  training  data  from  each  Soldier,  while  respecting  operator 
differences  in,  e.g.,  baseline  EEG  that  may  be  present  in  recordings  from  each  new  Soldier 
subject.  Applied  studies  under  TE  II  using  a  number  of  subjects  will  allow  early  investigation 
within  the  CTA  of  the  extent  to  which  individualized  data  models  perform  better  than  a  common 
data  model,  while  computational  approaches  to  characterizing  relationships  between  individual 
and  group  data,  developed  under  B.5  will  allow  first  testing  of  methods  to  quickly  individualize  a 
common  model  from  partial  (baseline)  EEG  data  on  each  individual. 

In  addition  to  the  scientific  barriers,  the  translational  process  will  have  to  address  technical 
challenges.  The  primary  technical  challenges  that  the  advanced  tools  for  brain  imaging  and 
neurocognitive  evaluation  TE  will  face  are  as  follows: 

•  Research  tools  must  be  understandable,  robust,  and  usable  by  personnel  with  basic  entry- 
level  programming  and  research  skills. 

•  High-density  EEG  collection  system  must  be  compatible  with  integration  into  current 
Army  equipment  in  a  manner  that  is  both  physically  robust  and  safe  for  the  user. 

•  Real-time  tools  must  be  optimized  for  speed  and  allow  system  developers  the  flexibility  to 
integrate  with  a  variety  of  system  configurations. 
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4.2.3  TE  II.  Transitional  Timeline 

Six  primary  Alliance  research  topic  areas  feed  into  this  TE: 

1.  We  will  develop  hardware  and  software  tools  to  collect,  synchronize,  and  organize  multiple 
data  streams  and  formats,  empowering  us  to  observe  and  record  a  variety  of  levels 
(physiological,  physical,  and  environmental)  of  neurobehavioral  data  regarding  operator 
and  environmental  state. 

2.  We  will  refine  and  validate  acquisition  hardware  and  related  tools  for  use  in  complex 
operational  environments. 

3.  The  Alliance  will  develop,  refine,  and  validate  software  tools  and  algorithms  for  the 
visualization,  mining,  and  analysis  of  the  complex,  high-density  data  streams  to  be 
obtained  through  CTA  efforts.  This  effort  is  a  most  critical  link  in  the  proposed  research, 
since  combining  brain,  body,  and  performance  measures  to  create  cognitive  monitoring 
models  is  both  highly  innovative  and  has  never  been  attempted  before,  thus  requiring  a 
strong  and  continued  focus  of  expertise  and  effort  by  several  CaN  CTA  computational 
investigators  working  in  concert  with  CaN  CTA  experimental  researchers. 

4.  The  acquisition,  reduction,  and  mining  tools  will  be  assessed  and  tuned  for  usability  and 
robust  application  across  experimental  tasks  and  in  a  variety  of  experimental  contexts. 

5.  Accompanying  and  following  the  tool  development,  a  fifth  area  for  this  TE  will  explore 
and  apply  data  acquisition,  reduction,  and  mining  tools  and  techniques  to  real-time 
systems.  This,  again,  is  a  critical  link  that  will  require  strong  innovation  and  continued 
collaboration  among  computational  and  experimental  CaN  CTA  researchers. 

6.  The  hardware  and  software  developed  for  implementation  in  real-time  systems  will  be 
assessed  and  further  matured  for  enhanced  usability  and  ruggedized  for  implementation  in 
Army  test-bed  systems. 

As  shown  in  figure  4,  there  will  be  two  main  transitions  that  result  from  efforts  in  the  six  focus 
areas.  Once  available,  the  software  and  hardware  tools  developed  in  the  first  four  focus  areas 
will  be  leveraged  into  DoD  programs  (outside  of  the  Alliance  collaborators)  that  aim  to  study 
and  use  information  regarding  Soldier  neurocognitive  performance  and  the  brain/behavioral 
dynamics  supporting  it.  Eikewise,  as  the  system  is  translated  into  a  set  of  robust  tools  and 
methods  for  real-time  use  through  efforts  in  the  fifth  and  sixth  focus  areas,  the  Alliance  will 
collaborate  to  facilitate  the  integration  of  the  neurotechnologies  within  Army  test-beds. 
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I  Alliance  Effort  B  Army  Program  Leveraging  Transition 


Figure  4.  Translational  timeline  depicting  expected  progress  in  TE  11. 

Advanced  tools  will  be  continually  developed,  refined,  and  matured  throughout  the  duration  of 
the  CTA  program.  In  the  first  year,  efforts  will  be  focused  on  primarily  in  three  key  areas: 

(1)  the  development  of  hardware  and  software  supporting  acquisition  of  high-density  data 
regarding  neurocognitive  performance,  (2)  the  development  of  a  central  database  structure  and 
format  for  archiving  data  from  all  CTA  research,  and  (3)  large-scale  data  mining  (enable  by  area 
2)  beginning  in  the  first  year  and  continuing  over  subsequent  years  as  data  are  generated  and 
integrated  into  the  database.  Developing  a  central  database  will  require  parallel  development  of 
a  cyber  infrastructure  for  integrating  joint  brain,  behavior,  and  environmental  data  and  analyzing 
multi-factorial  datasets  using  high-performance  computing.  We  will  also  develop  an 
experimental  real-time  interactive  control  and  analysis  (ERICA)  system  with  UCSD’s  DataRiver 
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system  (Vankov  et  al.,  2010)  for  distributed  data  aequisition,  synchronization,  online  processing, 
and  stimulus  delivery  in  the  first  year.  Initial  efforts  in  the  first  year  will  also  focus  on 
preliminary  real-time  analysis  techniques  and  tool  maturation  for  existing  toolboxes. 

From  year  2  onward,  the  acquisition,  reduction,  and  mining  tools  developed  will  be  applied, 
assessed,  and  refined  for  greater  integration  of  highly  multivariate,  multi-aspect  neurobehavioral 
data  as  well  as  later  progress  towards  application  in  real-time  systems.  As  shown  in  the  timeline 
in  figure  4,  much  of  the  development  for  this  TE  will  proceed  in  parallel,  with  data  acquisition 
feeding,  for  example,  mining  and  visualization  tools  that  will  likewise,  feed  information  back  for 
refinement  of  the  hardware  and  software  tools  underlying  the  data  acquisition  process  itself,  as 
well  as  experimental  design  features  that  may  be  required  to  test  and  extend  them. 

Similarly,  the  maturation  of  the  tools  for  usability  and  robustness  across  tasks  and  contexts  will 
enable  their  adoption  for  use  by  other  DoD  laboratories  that,  in  turn,  may  provide  feedback  and 
suggestions  for  subsequent  tool  refinement  and  maturation.  Research  in  from  the  third  year 
onward  focuses  on  exploiting  and  extending  the  developments  from  the  first  two  years  and 
exploiting  the  technologies  from  TE  I,  as  well  as  on  the  maturation  and  ruggedization  of  systems 
for  use  in  real-world  settings. 

These  efforts  are  described  in  the  Alliance  research  projects  supporting  this  transition,  which 
include  the  following: 

•  Brain  Dynamics  of  Attention  Shifting  (DAS) 

•  Experimental  Real-time  Control  (ERC) 

•  Data  Eusion,  Summary,  and  Visualization  (ESV) 

•  MBE 

•  Multi-Screen  Search  (MSS) 

•  Neurocomputation  (NCP) 

•  Speech  Comprehension  and  Neurolinguistics  (SCN) 

•  VME 

•  WDT 

In  addition  to  the  research  projects,  this  TE  will  require  additional  specific  efforts  to  address  the 
technical  challenges. 

•  Through  all  years,  as  tools  approach  completion,  ARE  will  lead  an  effort,  in  collaboration 
with  DCS  Corp,  to  evaluate  developed  algorithms,  interfaces,  and  hardware  for  general 
usability  and  appropriateness  for  research  laboratory  and  in-field  use.  In  the  initial  two 
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years,  this  effort  (lead  Chris  Stachowiak)  will  focus  on  the  usability  and  general  human 
factors  of  Matlab-based  software  tools  developed. 

Critical  Points 

There  are  five  critical  points  in  the  successful  completion  of  this  project: 

C.l.  The  required  specifications  for  the  data  acquisition  hardware,  software,  and  file  formats 
will  be  defined  and  agreed  upon.  Usability  standards  for  software  tools  will  be  described 
and  defined.  The  ERICA  software  system  will  be  made  available  by  Makeig,  Vankov 
and  colleagues  at  UCSD  for  use  by  Alliance  members. 

C.2.  Data  acquisition  and  management  infrastructure  will  be  fully  implemented  and  integrated 
with  a  centralized  database  for  all  Alliance  members  to  access  and  for  use  in  the 
Simulation  phase  of  the  experimentation.  Beta  phase  releases  of  a  subset  of  tools  for 
mining,  visualization,  and  analysis  will  be  made  available  for  Alliance  members  as  well 
as  other  DoD  laboratories.  Initial  scenario  development  will  be  complete  for  the 
simulation  experiments  and  scenarios  will  have  been  assessed  on  pilot  data. 

C.3.  All  data  will  be  collected  for  the  Simulation  phase  experiments  by  the  end  of  the  first 
quarter  of  year  4.  All  data  generated  during  the  Simulation  phase  experiments  will  be 
submitted  to  the  CaN  CTA  database  for  archival  as  well  as  analysis  activities.  New  and 
revised  requirements  and  specifications  for  the  CaN  CTA  tool  set  will  be  defined  and 
agreed  upon. 

C.4.  A  suite  of  CaN  CTA  tools  for  brain  imaging  and  neurocognitive  performance  assessment 
will  be  released  in  full  form,  following  integration  of  feedback  from  Alliance  members 
and  other  DoD  laboratories.  Real-time  acquisition,  reduction,  and  analysis  toolset  will  be 
delivered  for  integration  into  Army  test-bed. 

C.5.  A  full  suite  of  tools  for  both  online  and  offline  acquisition,  processing,  and  analysis  will 
be  delivered  to  and  integrated  with  Army  test-bed  systems. 

4.2.4  TE  II.  Transitions 

The  purpose  of  this  TE  is  to  provide  hardware  and  software  tools  that  will  enable  advanced 
monitoring  of  Soldier  performance  in  operational  environments  as  research  evolves  from 
laboratory  to  simulation  to  real-world  settings.  In  particular,  the  objective  of  this  TE  is  to 
provide  tools  necessary  for  the  implementation,  integration,  and  interpretation  of  MoBI  into  real- 
world  environments  and  the  integration  of  MoBI  into  DoD  systems.  More  broadly,  the 
transitioning  of  advanced  tools  for  the  acquisition,  reduction,  visualization,  and  mining  of  high- 
density  information  regarding  brain  and  behavioral  dynamics  will  enable  vast  changes  in  the 
ability  to  conduct  research  within  real-world,  operationally  relevant  environments.  However,  the 
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tool  developments  envisioned  to  be  advanced  within  this  TE  must  be  matured  before  longer-term 
transitions  can  be  realized.  For  example,  beyond  the  definition  and  development  of  hardware 
and  software  supporting  the  implementation  of  advanced  acquisition,  reduction,  visualization, 
and  mining  capabilities  is  the  enhancement  of  system  usability  and  robustness  for  breadth  of 
application  across  user  groups  and  testing/evaluation  environments,  including  ruggedization  for 
use  in  operational  environments.  Ultimately,  the  efforts  under  TE  II  will  lead  to  later  transitions 
influencing  the  advanced  design  of  crew  station  systems  and  real-time  Soldier  monitoring 
applications.  By  enabling  investigations  of  brain  function  in  complex  dynamic  environments, 
this  will  facilitate  discovery  of  previously  unexplored  relations  among  multiple  brain,  behavior, 
and  environmental  variables  targeting  new  insights  to  old  problems,  and  contribute  to  the 
advancement  of  theories  of  human  performance,  and  contribute  to  more  comprehensive  and 
accurate  models  of  Soldier  performance  that  will  better  serve  the  Army  across  a  variety  of  task 
domains. 

4.3  TE  III.  Enhanced  Design  of  High-information  Multichannel  Crew  Stations 

Advances  in  enabling  technologies  are  enhancing  Soldier-system  performance  and  expanding 
operational  capabilities;  however,  these  advances  are  also  intensifying  the  need  for  Soldiers  to 
operate  in  complex,  mixed-initiative  system  environments  in  which  dynamic,  complex 
information  streams  must  be  integrated  across  multiple  spatial  and  temporal  scales. 

Technological  advances,  particularly  in  sensor  deployment,  information  bandwidth,  and 
automation,  coupled  with  economic  and  political  realities,  continue  to  place  more  and  more 
responsibility  on  fast,  distributed,  and  effectively  independent  decisions  by  solo  or  small  groups 
of  Soldiers  who  control  ever  more  potent  defensive  and  offensive  assets.  Under  such 
circumstances.  Soldier  cognitive  failures  in  comprehension  and  decision  making  based  on  ever 
more  complex  data  streams  can  become  a  critical  bottleneck  in  Army  defensive  and  offensive 
capabilities.  The  CaN  CTA  research  program  aims  to  develop  neuroscience-based  principles 
that  focus  on  how  best  to  assist  people  to  operate  in  complex,  mixed-initiative  system 
environments  in  which  dynamic,  complex  information  streams  must  be  integrated  across 
multiple  spatial  and  temporal  scales.  This  TE  focuses  on  prototype  high-information 
multichannel  crew  stations  that  reflect  the  complex  socio-technical  environments  envisioned 
above.  Specifically,  the  TE  integrates  the  basic  science  and  neurotechnology  efforts  over  the 
course  of  the  CaN  CTA  into  a  prototype  Warfighter- machine  interface  (WMI)  to  enhance 
Soldier-system  and  team  performance. 

4.3.1  TE  III.  Background 

As  Future  Force  concepts  have  evolved  in  the  direction  of  employing  computationally  intelligent 
architectures  to  compliment  and  supplement  Soldier  abilities,  concepts  and  methods  regarding 
the  acquisition  and  use  of  information  have  become  increasingly  central.  The  focus  of  these 
concepts  demands  continued  attention  towards  improvement  of  the  means  and  methods  by  which 
information  is  both  transmitted  to  and  collected  from  the  Warfighters.  That  is,  as  a  focal  point  of 
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continued  efforts,  emphasis  on  the  interfaces  between  Warfighter  and  system  is  not  only 
warranted,  but,  in  fact,  is  a  critical  element  of  future-oriented  S&T  programs.  Through  a  variety 
of  programs,  the  U.S.  Army  has  made  considerable  investments  into  the  development  of  Soldier- 
machine  interfaee  technologies  to  support  the  conduct  of  military  operations.  With  research  and 
development  efforts  beginning  as  early  as  the  mid  1980s  (i.e.,  on  the  Inter- Vehicular  Information 
System  [IVIS]  fielded  in  the  M1A2  Abrams),  military  interface  technologies  have  evolved 
through  literally  deeades  of  research,  development,  and  engineering  evaluations  as  well  as 
through,  for  some  systems  (e.g.,  IVIS,  Force  XXI  Battle  Command,  Brigade-and-Below 
[FBCB2]),  use  in-theater  (Morrison  et  al,  2003).  Currently,  TARDEC,  along  with  its  partners, 
including  ARE,  the  U.S.  Communications-Electronics  Research  Development  and  Engineering 
Center  (CERDEC),  and  the  Night  Vision  and  Electronic  Sensors  Directorate  (NVESD),  is 
continuing  development  of  crew  station  solutions  for  WMI  technologies,  an  example  of  which  is 
shown  in  figure  5. 


Figure  5.  Example  of  crew  station  technology  developed  to  optimize  Soldier  workload  and 
provide  the  ability  to  conduct  mission  planning;  route  planning;  reconnaissance, 
surveillance,  and  target  acquisition  (RSTA);  and  fire  control  capabilities.  The  crew 
station  WMI  software  is  fully  configurable  (up  to  six  possible  independent  screens 
may  be  displayed  simultaneously),  portable  (between  crew  stations),  and 
interoperable.  Here,  two  crew  stations  are  shown  as  mounted  on  a  large  6  DOF 
motion  simulator  located  at  TARDEC. 


The  modem  direction  of  WMI  development  is  towards  highly  complex,  mixed-initiative  system 
environments.  Eor  example,  a  vision  of  the  future  environment  includes  a  highly  networked 
“family”  of  tactical  assets  including  both  manned  and  unmanned  ground  and  aerial  vehicles 
along  with  remote  sensing  and  weapons  capabilities.  In  this  vision,  the  WMI  is  to  provide  the 
ability  for  individual  Soldiers  to  have  access  to  broad-area  informational  resources  developed 
from  those  assets  in  order  to  perform  one  of  over  100  possible  predefined  roles.  Therefore, 
modularity  and  flexibility  of  interface  design  have  been  essential. 
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At  the  same  time,  however,  to  avoid  heavy  training  burdens,  it  has  been  seen  as  desirable  to  have 
a  WMI  that  shares  common  features  and  functionality  across  roles  and  tasks.  The  flexibility  of 
current  TARDEC-ARL  crew  station  solutions  results  in  tremendous  functionality  that  includes 
imagery  that  makes  relatively  large,  prominent  screen  space  claims,  (e.g.,  vehicle  teleoperation 
displays,  mission  planning,  vehicle  sensor  controls,  local  area  and  3-D  situational  awareness 
monitoring,  multiple-vehicle  formations  control,  robotics  control  asset  list)  and  secondary 
functions  that  can  be  minimized  to  preserve  screen  space  (i.e.,  asset  maps,  route  plan  maps,  and 
warnings,  cautions,  and  alerts  [WACAs]). 

Beyond  human  factors  considerations  that  have  driven  the  screen  design  concepts,  layout,  and 
functionality  built  into  the  current  TARDEC-ARL  crew  station  solutions,  the  CaN  CTA  research 
will  enable  systems  designers  to  address  a  variety  of  critical  high-level  functions  that  have  yet  to 
be  fully  and  successfully  addressed  by  enabling  technologies  integrated  with  prior  WMIs. 

Secure  mobility 

Secure  mobility  is  a  concept  that  was  created  to  help  capture,  in  a  descriptive  sense,  the  key 
differences  in  the  management  of  vehicle  mobility  between  civilian  and  military  task  contexts 
(McDowell,  Oie,  Tierney,  and  Elascher,  2007;  McDowell,  Nunez,  Hutchins,  and  Metcalfe, 

2008).  In  the  civilian  context,  the  goals  associated  with  driving  include,  at  a  high  level,  path 
selection,  position  maintenance,  and  safety  (collision  avoidance)  for  those  inside  the  vehicle  as 
well  as  those  outside  (pedestrians,  other  vehicles).  In  the  military  context,  however,  in  addition 
to  the  concerns  associated  with  basic  driving  and  navigation.  Soldiers  have  the  added  expectation 
of  maintaining  local  area  awareness  sufficient  to  detect  threats  to  their  own  safety  and  that  of 
their  platoon,  as  well  as  any  semiautonomous  (robotic)  assets  that  they  may  be  within  their 
purview.  Therefore,  there  are  added  demands  and  dynamics  to  military  mobility  tasks  that  do 
not  exist  in  standard  civilian  driving  conditions.  Eurther,  an  important  contribution  to  the 
cognitive  dynamics  of  secure  mobility,  as  it  is  typically  executed  under  current  force  operations, 
is  that  of  team  function.  That  is,  because  of  the  complexity  and  level  of  threat  associated  with 
military  environments,  secure  mobility  is  currently  treated  as  a  team  function  in  that  one  Soldier 
monitors  navigation,  another  drives  the  vehicle,  and  yet  others  serve  as  look-outs  responsible  for 
different  sectors  of  observation  across  the  360°  surround.  Tools  and  techniques  are  thus  needed 
to  enhance  abilities  to  efficiently  and  effectively  maintain  secure  mobility  during  military 
operations. 

Complex  and  Multitasking  Environments. 

Indeed,  military  environments  are  highly  complex  and  dynamic.  More  to  the  point,  the  range  of 
variability  associated  with  military  operational  environments  is  quite  vast.  Operational  tempo 
(OPTEMPO),  for  example,  can  vary  from  extremely  slowly  changing  dynamics,  such  as  in 
scenarios  involving  the  need  to  manage  persistent  stare  over  extremely  protracted  periods  of  time 
(i.e.,  surveillance  over  a  potential  enemy  base  for  days  on  end),  to  extremely  rapidly  developing 
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dynamics,  such  as  in  the  midst  of  an  armed  confrontation  with  enemy  combatants  intermingled 
with  civilian  bystanders.  Of  course,  OPTEMPO  is  but  one  of  many  variable  factors  that 
influence  the  technologic  needs  of  the  Soldier  situated  in  theater;  variable  constraints  affecting 
communications,  visualization,  cognitive,  and  physical  fatigue;  maintenance  of  a  common 
operating  picture;  and  so  on  limit  the  abilities  of  the  individual  Soldier  in  the  modem  threat 
environment  as  well  as  drive  the  need  for  technologic  enhancements  to  the  WML  Beyond  the 
multimodal  complexity  affecting  mission  dynamics  is  the  fact  that  the  modern  Soldier  will  be 
expected  to  function  adaptively  under  the  pressures  of  maintaining  performance  on  multiple, 
temporally  overlapping  and,  in  some  cases,  competing  tasks — each  with  their  own  variable 
dynamics.  Examples  of  overlapping  tasking  may  include  monitoring  of  several  (intra-  and  inter- 
vehicular)  communications  streams,  monitoring  and  control  of  one  or  more  unmanned  tactical 
assets  (both  aerial  and  ground  assets),  and  monitoring  streams  of  (mostly  visual)  data  forming 
the  common  operating  picture,  along  with  other  RSTA  activities. 

Platoon  Leader  (PL)  Decision-Making 

In  the  U.S.  Army,  a  mechanized  platoon  of  light  or  heavy  armored  vehicles  can  consist  of 
number  of  Soldiers  and  a  variety  of  equipment.  An  Ml  tank  platoon  can  typically  contain  four 
tanks  with  four  crew  members  in  each  tank  (tank  commander,  gunner,  driver,  and  loader).  One 
of  the  tank  commanders  would  also  be  the  leader  of  the  platoon.  A  platoon  of  mechanized 
infantry  in  Bradley  Eighting  Vehicles  would  contain  four  vehicles  with  10  Soldiers  in  each. 

Some  of  the  Bradley  platoon  Soldiers  will  have  mounted  roles  and  some  will  have  dismount 
roles,  but  all  will  be  commanded  by  the  platoon  commander.  Such  platoon  organizations  are 
further  grouped  into  companies  made  up  of  three  to  four  platoons  with  additional  staff  and 
equipment  for  company  Headquarters.  Therefore,  decisions  to  be  made  at  the  PE  command 
echelon  affect  and  are  affected  by  both  top-down  and  bottom-up  elements  of  the  command 
structure.  In  order  to  make  adequate  decisions  at  the  PE  level,  one  must  be  aware  of  the  state  of 
subordinate  troops,  the  local  operational  environment,  the  state  of  the  company,  as  well  as  orders 
and  constraints  communicated  downstream  from  higher  echelons.  The  WMI,  therefore,  must 
also  provide  sufficient  capability  to  simultaneously  monitor  all  relevant  aspects  of  the 
informational  environment.  Given  the  complexity  of  the  typical  military  operational  context, 
information  presented  to  the  PE  should  not  be  presented  in  the  form  of  raw  data  (i.e.,  raw  video 
screens,  unfiltered  audio,  etc.),  but  rather  should  be  processed,  summarized,  and/or  enhanced  to 
facilitate  immediate  comprehension  for  the  sake  of  time-pressured  decision-making. 

Team  Dynamics/Cooperation 

As  with  secure  mobility  discussed  previously,  many  military  tasks  conducted  in  the  operational 
environment  involve  a  division  of  roles  with  an  aim  towards  integration  of  outcome.  That  is, 
military  tasks  are  commonly  subdivided  into  smaller  sets  of  roles  to  be  assigned  to  different 
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members  of  a  platoon  or  vehicle  crew  under  the  concept  that  as  each  Soldier  fulfills  his  or  her 
role,  the  overall  mission  will  be  brought  to  completion.  What  this  presumes,  however,  is  that  all 
members  of  the  team/platoon/squad  share,  in  continuous  real-time,  the  same  essential  cognitive 
ingredients  (i.e.,  information,  attention,  motivation,  intent,  etc.)  for  assembling  the  necessary 
orderly  plan  of  action  for  completing  the  assigned  task  or  tasks.  Moreover,  to  add  to  the 
complexity  of  cooperative  tasks,  certain  military  concepts  assume  that  tasks  broken  into 
subcomponents  in  this  way  can  be  completed  by  teams  of  non-co-located  Soldiers  (e.g.,  target 
acquisition  and  engagement).  Thus,  differing  perspectives/frames  of  reference  on  the  problem 
environment  as  well  as  the  dynamics  of  potentially  imperfect  communications  streams  have  the 
potential  to  alter  (or  at  least  dramatically  complicate)  the  ability  of  such  teams  to  operate 
effectively  and  efficiently  when  conducting  distributed  team  actions. 

Brain-system  Interface 

Finally,  the  ideal  concept  for  “closing  the  loop”  between  Soldier  and  system  through  a  well- 
designed  interface  is  to  provide  a  means  of  feeding  data  regarding  the  neurocognitive  state  of  the 
Soldier  into  the  WMI  that  is  being  used.  Such  an  achievement  would  be  a  major  advance  over 
current  action-based  (e.g.,  button  pressing)  methods  of  providing  Soldier  feedback  to  the  system 
function  as  it  would  not  rely  on  awaiting  appropriate  overt  action  from  the  Soldier  alone;  the 
system  could  function  based,  in  part,  on  the  neurocognitive  data  that  it  received  from  monitors  of 
Soldier  physiology  and  behavior.  This  inherently  suggests  that  data  regarding  the  moment-to- 
moment  dynamics  of  brain  function  (i.e.,  “brain  waves”  or  EEG  patterns)  should  be  integrated 
with  WMI  function.  However,  under  the  concepts  of  MoBI  as  discussed  earlier,  a  true  “brain 
system  interface”  must  include  more  than  EEG  data;  it  must  also  include  data  regarding  the 
behavior  that  is  being  optimized  and  controlled  by  brain  function.  Therefore,  information 
regarding  Soldier/operator  behavior  and  appraisal  of  the  stimulus  environment  to  be  gained  from 
measurement  of  gaze  point  location,  bodily  action,  and  autonomic  function  (e.g.,  heart  rate, 
respiration)  should  be  integrated  with  the  WMI  in  addition  to  EEG  for  the  efficient  leveraging  of 
neurocognitive  signatures  for  enhancing  overall  system  performance. 

4.3.2  TE  III.  Transition  Barriers  and  Hurdles 

As  already  established,  the  development  of  Soldier-machine  interface  technology  spans  several 
decades  of  science  and  engineering  efforts.  Starting  as  early  as  the  1980s,  military  interests  have 
funded  the  development  of  technologies  to  provide  battlespace  information  to  Soldiers  that  are 
located  within  vehicles  designed  to  simultaneously  enhance  both  survivability  and  lethality.  In 
the  CaN  CTA,  however,  efforts  are  planned  to  take  a  new  direction  in  enhancing  previous  WMI 
design  concepts  for  modem-era  military  use.  That  is,  the  research  and  development  efforts 
within  the  CaN  CTA  will  build  on  previous  WMI  concepts  to  integrate  the  Soldier  and  the 
system  in  ways  that  have  not  yet  been  achieved  for  fielded  systems.  The  high-information, 
multichannel  crew  station  to  be  prototyped  through  the  CaN  CTA  will  be  among  the  first  of  its 
kind  to  place  Soldier  and  system  on  equal  footing  as  members  of  the  same  team — with  the 
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system  leveraging  “knowledge”  regarding  the  state  of  the  Soldier  to  take  action  that  facilitates 
the  Soldiers  ability  to  complete  operational  tasks  both  efficiently  and  effectively.  In  order  to 
develop  such  a  prototype  system,  then,  this  preplanned  TE  will  have  to  build  from  the  early 
progress  made  in  crew  station  interface  design  in  addressing  all  five  primary  scientific  barriers: 

B.lThe  restriction  of  experimental  designs  to  highly  controlled  and  impoverished 
stimulus/response  paradigms  and  environments. 

Experimental  environments  must  advance  beyond  those  typically  developed  for  highly  controlled 
and  impoverished  laboratory  settings.  WMIs  developed  need  to  be  matured  and  validated  as 
robust  enough  for  application  in  highly  complex  environments,  multi-tasking  scenarios,  or  for 
operation  in  multiple  or  fluctuating  performance  domains  such  as  PE  decision  making  and  secure 
mobility. 

B.lThe  lack  of  portable,  user-acceptable  (e.g.,  comfortably  wearable),  and  robust  systems 
for  monitoring  brain  and  body  dynamics. 

Multi-purpose  acquisition  systems  need  to  able  to  collect  high-fidelity  EEG  data  robustly  and 
continuously  without  the  use  of  conductive  gels  or  skin  preparation.  Eurther,  portable  systems 
must  robustly  handle  the  expected  substantial,  in-vehicle  electrical  and  motion  artifacts  that  pose 
threats  to  reliable  and  valid  recording  of  brain  dynamics  in  operational  environments. 

B.  3  The  failure  to  record  the  whole  of  physical,  mental,  and  physiological  behavior  that  the 
brain  controls,  and  the  physical  and  socio-cultural  effects  of  the  environment  that  impact 
brain  function,  in  sufficient  detail  and  across  a  sufficient  breadth  of  circumstances. 

Data  acquisition  systems  must  be  generic  or  adaptable  enough  to  collect  data  in  the  sufficient 
detail  and  across  a  breadth  of  circumstances  and  in  conjunction  with  a  variety  of  interface 
configurations  supporting  a  number  of  military  roles.  Such  systems  must  incorporate  a  wide 
range  of  sensor  technologies  and  thus  must  address  issues  associated  with  sampling  rate, 
synchronization,  data  storage,  subsequent  data  retrieval,  and  overall  data  flow  between  system 
components.  Specifically,  multi-aspect  datasets  collecting  high-density  data  from  differing  types 
of  physiological  sources  (e.g.,  EEG,  EMG,  eye  and  body  movements)  are  often  very  large  and 
unwieldy  in  raw  form. 

B.4The  lack  of  mathematical  modeling  methods  and  software  to  find  statistical  relationships 
between  moment-to-moment  variations  in  environmental,  behavioral,  and  functional 
brain  dynamic  recordings. 

Due  to  the  novelty  and  the  scale  of  the  large,  multi-aspect  datasets  expected  from  within 
operationally  relevant  scenarios,  new  tools  for  data-reduction,  mining,  and  visualization  must  be 
developed.  In  particular,  as  systems  progress  towards  real-time  implementations,  such  tools 
must  enable  developers  to  constrain  system  designs  to  utilize  only  the  necessary  and  sufficient 
data  as  opposed  to  online  mining  and  processing  of  overwhelming  amounts  of  data.  Moreover, 


44 


in  order  for  such  information  to  be  most  useful  in  the  crew  station  context,  adequate  and 
advanced  methods  for  summarizing  complex  data  need  to  be  explored  alongside  methods  for 
efficient  and  effective  communication  of  complex  multichannel  (visual,  auditory,  text-based) 
data  streams. 

B.SThe  lack  of  sufficient  data  archives  and  resources  to  systematically  study  relationships 
between  individualized  models  derived  for  cognitive  monitoring  and  individual 
differences  in  performance,  cognitive  ability,  and  personality,  relationships  that  could 
optimize  performance  of  cognitive  monitoring  systems  while  lowering  requirements  for 
collecting  individual  subject  training  data. 

The  WMI  designed  and  tested  under  TE  III  should  be  able  to  use  models  of  brain/behavioral 
changes  that  do  not  require  extensive  pilot  training  data  from  each  Soldier,  while  respecting 
operator  differences  in,  e.g.,  baseline  EEG  that  may  be  present  in  recordings  from  each  new 
Soldier  subject.  Applied  studies  under  TE  III  using  a  number  of  subjects  will  allow  early 
investigation  within  the  CTA  of  the  extent  to  which  individualized  data  models  perform  better 
than  a  common  data  model,  while  computational  approaches  to  characterizing  relationships 
between  individual  and  group  data,  developed  under  B.5  will  allow  first  testing  of  methods  to 
quickly  individualize  a  common  model  from  partial  (baseline)  EEG  data  on  each  individual. 

In  addition  to  the  scientific  barriers,  the  translational  process  will  have  to  address  technical 
challenges  related  to  the  specific  operational  topics  discussed  above.  The  primary  technical 
challenges  that  the  enhanced  design  of  high-information  multichannel  crew  stations  TE  will  face 
are  as  follows: 

•  Crew  station  interfaces  must  provide  data  and  visualization  tools  sufficient  for  maintenance 
of  real-time  local  area  awareness  during  vehicle  mobility  control.  Such  tools  may  include, 
but  are  not  necessarily  limited  to,  those  supporting  semi-autonomous  navigation,  which  has 
the  potential  to  alleviate  demands  on  driver  perception  and  attention  during  execution  of 
simultaneous  mobility  and  security  tasks  (McDowell  et  ah,  2008)  so  as  to  enable  execution 
of  secure  mobility  functions. 

•  Crew  station  interfaces  must  provide  communications  tools  sufficient  for  maintenance  of 
real-time  situational  awareness  during  multitasking  and  teaming/cooperation.  In  both 
multitasking  and  teaming/cooperation  scenarios,  maintenance  of  situational  awareness  is 
essential — allowing  the  operator  to  construct  and  hold  mental  representations  of  task  status 
during  execution  of  potentially  discontinuous,  irregular  OPTEMPO  operations  under 
constraints  of  partial  information  feedback. 

•  A  key  aspect  of  decision  making  is  the  ready  access  to  the  information  necessary  for 
arriving  at  an  optimal  choice  among  a  limited  set  of  viable  operational  options.  Therefore, 
to  support  decision  making,  both  advanced  intelligent  architectures  (including  autonomous 
agents)  and  real-time  communications  and  visualization  enhancements  will  necessarily 
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need  to  be  integrated.  The  combination  of  enhanced  informational  systems  supporting 
real-time  situational  awareness  with  intelligent  architectures  designed  to  assist  decision 
making  (e.g.,  an  “expert  system”  that  summarizes  data  and  prioritizes  potential  actions), 
especially  as  complimented  by  integrated  data  regarding  the  neurocognitive  states  of  each 
Soldier  in  a  PL’s  command,  has  considerable  potential  to  ease  and  economize  decision 
making  when  in  time-pressured  operational  circumstances. 

4.3.3  TE  III.  Translational  Timeline 

As  shown  in  figure  6,  seven  primary  Alliance  research  topic  areas  feed  into  this  TE: 

1.  First,  it  will  be  necessary  to  integrate  the  novel  data  acquisition,  reduction,  and  mining 
tools  from  TE  II  into  the  crew  station  as  they  become  available  through  CaN  CTA  efforts. 
The  existence  and  integration  of  such  tools  for  assessment  and  characterization  of  Soldier 
neurocognitive  state  and  performance  will  serve  as  a  foundation  on  which  progress  in  the 
remaining  focus  areas  will  rest. 

2.  We  will  develop  and  integrate  tools  to  advance  Soldier  capabilities  to  execute  secure 
mobility  functions  using  the  WML 

3.  In  a  parallel  effort,  the  CaN  CTA  will  develop  and  assess  new  WMI  technologies  for 
enhanced  performance  in  complex,  multitasking  environments  (including  secure  mobility 
functions). 

4.  In  a  similar  parallel  effort,  the  Alliance  will  address  the  challenge  of  providing 
technologies  to  enhance  PE-level  decision  making  in  operational  environments. 

5.  We  will  examine  communications  tools  for  facilitating  team  performance  during  execution 
of  cooperative  tasks. 

6.  The  Alliance  will  work  to  integrate  the  above  five  efforts  with  one  aimed  at  developing  and 
refining  MoBI-derived  brain-machine  interfaces  for  enhancement  of  Soldier-system 
performance  using  neurocognitive  data. 

7.  During  the  fifth  program  Year,  all  of  the  above  described  efforts  will  culminate  in  an  area 
of  focus  that  will  develop  a  prototype,  proof-of-principle  WMI  that  integrates 
neuroscience-based  design  elements,  or  neurotechnologies. 

There  will  be  one  main  transition  that  results  from  efforts  in  the  seven  focus  areas,  that  is,  the 
tools,  techniques,  and  technologies  developed  through  CaN  CTA  efforts  in  general,  as  well  as 
from  TE  III  in  particular,  will  result  in  a  prototype  high-information  multichannel  crew  station  to 
be  integrated  with  an  Army  vehicle  test-bed  system.  The  multichannel  crew  station  will  support 
advanced  visual  displays,  audio  and  text-based  communications,  brain-machine  interfaces,  and 
automation  for  enhancing  Soldier-system  performance  in  the  areas  of  secure  mobility, 
multitasking  support,  teaming/cooperative  mission  execution,  and  decision  making. 
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Figure  6.  Translational  timeline  depicting  expected  progress  in  TE  111. 

The  high-information  multichannel  crew  station  TE  illustrates  a  progressive  process  to  be 
metered  out  across  the  duration  of  the  CaN  CTA  program  period.  In  year  1,  the  focus  in  TE  III 
will  be  on  the  integration  of  advanced  tools  from  TE  II  as  they  are  matured  to  the  point  where 
they  may  be  used  in  the  crew  station  environment.  Year  2  will  then  begin  focus  on  applying  the 
advanced  tools  to  the  high-level  objectives  of  establishing  tools  for  facilitating  goals  associated 
with  secure  mobility  as  well  as  enabling  performance  in  complex,  multitasking  environments. 
Year  3  onward  will  then  see  the  enhancement  of  the  crew  station  environment  and  associated 
tools  for  use  in  team  dynamics  and  decision  making,  as  well  as  integration  of  MoBI-derived  tools 
for  brain-system  interfacing.  Eurther,  from  year  3  onward,  efforts  will  aim  towards  integration 
of  tools  and  techniques  with  crew  stations  in  Army  test-bed  environments.  In  the  final  year, 
proof-of-principle  neurotechnologies  will  be  integrated  to  complete  the  final  transition  of  a  high- 
information  multichannel  crew  station  prototype  for  use  in  complex  military  environments. 
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These  efforts  are  described  in  the  Alliance  research  projects  supporting  this  transition,  which 
include  the  following: 

•  The  Construction  and  Use  of  Cognitive  Maps  Under  Threat  (CMT) 

•  DAS 

•  Constructing  Mutually-derived  Situational  Awareness  via  EEG-informed  Graph-based 
Transductive  Inference  (GTI) 

•  MBE 

•  Modeling  Perceptual  Decision  Making  (MDM) 

•  Multi-Modal  Sensory  Attention  (MSA) 

•  MSS 

•  Using  Neural  Biomarkers  to  Predict  and  Guide  an  IndividuaPs  Ability  to  Optimally  Adapt 
Decision  Rules  (NBD) 

•  Optimal  Decision  Making  (ODM) 

•  ODE 

•  Using  Neural  Activity  to  PAE 

•  Perceptual  Integration  Across  Time  and  Modality  (PTM) 

•  SCN 

In  addition  to  the  research  projects,  this  TE  will  require  additional  specific  efforts  to  address  the 
technical  challenges. 

•  Erom  Year  3  onward,  as  tools  approach  completion  ARE  will  lead  an  effort,  in 
collaboration  with  DCS  Corp,  to  evaluate  developed  tools,  techniques,  and  interface 
technologies  for  general  usability  and  appropriateness  for  research  laboratory  and  in-field 
utilization. 

Critical  Points 

There  are  five  critical  points  in  the  successful  completion  of  this  project. 

C.  1  The  ERICA/DataRiver  data  acquisition  system  will  undergo  initial  integration  with 

ARE/TARDEC  crew  station  test-bed.  In  particular,  efforts  will  focus  on  issues  affecting 
synchronization  and  storage  of  high-density,  multimodal  data  to  be  obtained  using 
paradigms  developed  under  the  MoBI  concept. 
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C.2.  A  preliminary  scenario  concept  will  be  developed  for  assessment  of  crew  station 
functionality  in  final  capstone  phase  experiments.  A  gap  analysis,  focusing  on 
discrepancies  between  simulation  phase  and  final  capstone  phase  experimental  scenarios 
will  be  completed. 

C.3.  Gaps  between  simulation  phase  and  final  capstone  phase  experimental  scenarios  will  be 
investigated.  Principles  and  concepts  for  implementation  in  neurotechnologies  proof-of- 
principle  will  be  identified  and  development  requirements  will  be  specified. 

C.4.  Scenarios  for  final  capstone  phase  experiment  will  be  finalized  and  piloted.  Brain-system 
interfaces  designed  for  integration  with  the  proof-of-principle  neurotechnologies  will  be 
available. 

C.5.  Test  and  evaluation  of  the  neurotechnology  prototypes  in  crew  station  test-bed  will  take 
place.  All  prototype  hardware/software  and  results  will  be  delivered  to  the  transition 
customer. 

4.3.4  TE  III.  Transitions 

The  objective  of  this  TE  is  to  develop  and  deliver  a  prototype  crew  station  system  that  will 
advance  WMI  technologies  beyond  those  designed  under  traditional  systems,  software,  and 
human  factors  engineering  concepts.  Specifically,  as  systems  have  been  typically  developed, 
end  users  have  only  been  considered  during  system  design  at  two  levels:  (1)  when  specifying  the 
goal  state  for  the  system  in  terms  of  user  functionality  and  (2)  in  designing  the  interface  once  the 
system  and  its  functionality  have  been  defined  and  largely  developed.  Yet,  in  terms  of  the  actual 
mechanisms  of  system  function,  generally  speaking,  the  needs  of  (and  potential  contributions 
from)  the  user  have  largely  been  subordinated  to  the  engineering  challenges  encountered  during 
system  design  and  development.  From  a  neuroergonomic  standpoint,  however,  it  is  considered 
to  be  advantageous  to  account  for  users  and  their  functional  properties  (i.e.,  neurocognitive 
function) /rom  the  outset  of  and  throughout  the  process.  As  such,  the  goal  of  this  TE  is  to 
leverage  tools  (hardware  and  software)  and  techniques  as  developed  through  CaN  CTA  research 
efforts  in  general,  as  well  as  from  the  other  two  TEs  in  particular,  to  enhance  performance  on  a 
selected  set  of  functions  that  include  secure  mobility,  complex  and  multitasking  environments, 
PE  decision  making,  and  team  dynamics  and  cooperation.  Such  technologies  for  leveraging 
aspects  of  neurocognitive  function  will  be  realized  through  brain-system  interfaces  that  enable 
proof-of-principle  neurotechnologies  to  be  instantiated  in  an  Army  test-bed  system.  By 
enhancing  the  design  of  WMIs  in  this  fashion,  it  is  expected  that  the  activities  executed  under 
this  TE  will  bring  future  force  concepts  closer  to  their  ultimate  realization  in  crew  station 
systems. 
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Appendix.  Listing  of  Projects  and  Principal  Investigators 

Table  A-1  is  a  list  of  the  projects  and  their  principal  investigators. 

Table  A-1.  Projects  and  principal  investigators. 

Project  Title 

Project 

ID 

Principal  Investigators 

Project  Keywords 

TAl:  Neurocognitive  Performance 

Ambulatory 

Neuroergonomics 

ANE 

Dr.  Dan  Perris,  U.  Michigan 

Dr.  Jason  Metcalfe,  DCS  Corp 

Locomotion,  Motor  control, 
Lower-limb  dynamics 

Construction  (&  Use  of 
Cognitive  Maps  Under 
Threat 

CMT 

Dr.  Chin-Teng  Lin,  NCTU 

Dr.  Peter  Konig,  U.  Osnabriick 

Spatial  cognition;  Spatial 
navigation;  Multisensory 
integration; 

Large-Scale  Experiments 

LSE 

Dr.  Scott  Makeig,  UCSD 

Attentional  Switching, 
Multitasking,  Individual 
Differences 

Modeling  Perceptual 
Decision  Making 

MDM 

Dr.  Angela  J.  Yu,  UCSD 

Cognitive  modeling, 

Individual  differences, 

Bayesian  predictive  models, 
Perceptual  contingency 
learning 

Multi-modality  Sensory 
Attention 

MSA 

Dr.  Peter  Konig,  U.  Osnabriick 

Multisensory  integration, 
Attention  allocation.  Tactile 
displays 

Multi-screen  Search 

MSS 

Dr.  Klaus  Gramann,  U.  Osnabriick 
Dr.  Jason  Metcalfe,  DCS  Corp. 

Visual  search.  Internal 
reference  frames,  Multi-modal 
sensory  inputs,  Situational 
awareness 

Using  Neural  Activity  to 
Predict  and  Guide  an 
Individual’s  Ability  to 
Optimally  Adapt  Decision 
Rules  * 

NBD 

Dr.  Michael  Miller,  UCSB 

Dr.  Barry  Giesbrecht,  UCSB 

Dr.  Scott  Grafton,  UCSB 

Dr.  Jean  Vettel,  ARL 

Signal  detection  theory, 

Optimal  decision  making, 
Individual  differences 

Operator  Dynamics  of 

Event  Appraisal 

ODE 

Dr.  Jason  Metcalfe,  DCS  Corp 

Dr.  Stephen.  Gordon,  DCS  Corp 

Dr.  W.  David  Hairston,  ARL 

Dr.  Will  Nothwang  ARL 

Event  appraisal;  Pacial 
expression;  Affective 
response;  Multisensor  fusion 

Using  Neural  Activity  to 
Predict  Attentional  Lapses 
During  Multisensory  Task 
Scenarios* 

PAL 

Dr.  Daniel  Weissman,  U.  Michigan 
Dr.  Barry  Giesbrecht,  UCSB 

Dr.  W.  David  Hairston,  ARL 

Machine  learning, 

Performance  prediction, 
Attentional  lapses 

Speech  Comprehension 
and  Neurolinguistics 

SCN 

Dr.  Roger  Levy,  UCSD 

Semantic  processing, 
Neurolinguistics 

*-  Year  2  Seedling  award.  Seedling  projects  are  awarded  based  on  competitive  solicitation  to  address  specific  issues  identified  as 
priorities  for  the  Army.  Project  proposals  can  be  up  to  three  years  in  duration,  and  are  reviewed  annually. 
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Table  A-1.  Projects  and  principal  investigators  (continued). 


Project  Title 

Project 

ID 

Principal  Investigators 

Project  Keywords 

TA2:  Neurocomputation 

Connectome-based 
Advancement  of  Brain 
Analysis* 

CBA 

Dr.  Walter  Schneider,  U.  Pitt 

Dr.  Don  Krieger,  U.  Pitt 

Dr.  Jean  Vettel,  ARE 

Individual  differences. 

Diffusion  weighted  imaging, 
Structure-function  coupling 

Dynamics  of  Attention 
Shifting 

DAS 

Dr.  Scott  Makeig,  UCSD 

Dr.  Jeanne  Townsend,  UCSD 

Selective  attention.  Cortical 
dynamics 

Data  Fusion,  Summary,  and 
Visualization 

ESV 

Dr.  Kay  Robbins,  UTSA 

Dr.  Nandini  Kannan,  UTSA 

Dr.  Yufei  Huang,  UTSA 

Data  visualization.  Database 
design 

Neurocomputation 

NCP 

Dr.  Kenneth  Kreutz-Delgado,  UCSD 
Dr.  Roger  Levy,  UCSD 

Dr.  Angela  J  Yu„  UCSD 

Dr.  Scott  Makeig,  UCSD 

Independent  component 
analysis  (ICA),  Statistical 
modeling.  Adaptive  Mixture 

ICA  (AMICA) 

*-  Year  2  Seedling  award.  Seedling  projects  are  awarded  based  on  competitive  solicitation  to  address  specific  issues  identified 
as  priorities  for  the  Army.  Project  proposals  can  be  up  to  three  years  in  duration,  and  are  reviewed  annually. 

TA3:  Neurotechnologies 

Constructing  Mutually- 
derived  Situational 

Awareness  via  EEG- 
informed  Graph-based 
Transductive  Inference* 

GTI 

Dr.  Paul  Sajda,  Columbia  U. 

Dr.  Brent  Lance,  ARE 

Rapid  Serial  Visual 

Presentation, 

MoBI-based  Eeedback  for 
Training  of  Automated 
Systems  and  System 
Operators 

MBE 

Dr.  Scott  Makeig,  UCSD 

Brain-computer  interaction. 
Adaptive  systems,  Training 

Effects  of  Vehicle  Motion 
and  Cognitive  Eatigue 

VME 

Dr.  Chin-Teng  Lin,  NCTU 

Dr.  Tzyy-Ping  Jung,  UCSD 

Brain-computer  interaction. 

User  monitoring.  Data 
acquisition.  Real-time  data 
processing 

Wearable  EEG 

Development  and  Testing 

WDT 

Dr.  Chin-Teng  Lin,  NCTU 

Dr.  Tzyy-Ping  Jung,  UCSD 

Wearable  systems.  Sensor 
development 

*-  Year  2  Seedling  award.  Seedling  projects  are  awarded  based  on  competitive  solicitation  to  address  specific  issues  identified  as 
priorities  for  the  Army.  Project  proposals  can  be  up  to  three  years  in  duration,  and  are  reviewed  annually. 
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List  of  Symbols,  Abbreviations,  and  Acronyms 

3-D  three-dimensional 

AgCl  silver  chloride 

ARL  U.S.  Army  Research  Laboratory 

BCI  brain-computer  interface 

CaN  Cognition  and  Neuroergonomics 

CTA  Collaborative  Technology  Alliance 

CERDEC  U.S.  Army  Communications-Electronics  Research,  Development  and  Engineering 
Center 

CES  Command  Environment  Simulator 

CMT  Cognitive  Maps  Under  Threat 

DAS  Brain  Dynamics  of  Attention  Shifting 

DCS  DCS  Corporation 

DOD  Department  of  Defense 

DoE  degrees  of  freedom 

ECG  electrocardiography 

EEC  electroencephalograM 

EMG  electromyography 

ERC  Experimental  Real-time  Control 

ERICA  Experimental  Real-time  Interactive  Control  and  Analysis 

EBCB2  Eorce  XXI  Battle  Command,  Brigade-and  Below 

fMRI  Eunctional  Magnetic  Resonance  Imaging 

ESV  Data  Eusion,  Summary,  and  Visualization 

GI  Graph  Invariant 

GTI  Graph-based  Transductive  Infrerence 

IC  independent  component 
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ICA 

lED 

IVIS 

MBF 

MDP 

MEMS 

MoBI 

mocap 

MOM 

MR 

MSA 

MSS 

NBD 

NCP 

NCTU 

NVESD 

ODE 

ODM 

PAE 

PET 

PE 

PTM 

RSTA 

S&T 

SBE 

SON 


Independent  Component  Analysis 
improvised  explosive  device 
Inter- Vehicular  Information  System 

MoBI-Based  Feedback  for  Training  Automated  Systems  and  System  Operators 

Markov  Decision  Process 

microelectromechanical  system 

mobile  Brain/Body  Imaging 

motion  capture 

Modeling  Perceptual  Decision  Making 
magnetic  resonance  (imaging) 

Multi-modal  Sensory  Attention 
Multi-Screen  Search 

Using  Neural  Biomarkers  to  Predict  and  Guide  an  Individual’s  Ability  to 

Optimally  adapt  Decision  Rules 

Neurocomputation 

National  Chiao  Tung  University 

Night  Vision  Electronic  Sensors  Directorate 

Operator  Dynamics  of  Event  Appraisal 

Optimal  Decision  Making 

Predict  Attentional  Eapses  During  Multisensory  Task  Scenarios 
Positron  Emission  Tomography 
platoon  leader 

Perceptual  Integration  Across  Time  and  Modality 
reconnaissance,  surveillance,  and  target  acquisition 
science  &  technology 
sparse  Bayesian  learning 
Speech  Comprehension  and  Neurolinguistics 
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TA 

TARDEC 

TBL 

TE 

UCSB 

UCSD 

UTSA 

VME 

VR 

WACA 

WDT 

WMI 


Technical  Area 

U.S.  Army  Tank  Automotive  Research  Development  and  Engineering  Center 
tramatic  brain  injury 
Transition  Effort 

University  of  California  Santa  Barbara 
University  of  California  San  Diego 
University  of  Texas  San  Antonio 
Effects  of  Vehicle  Motion  and  Cognitive  Eatigue 
virtual  reality 

warnings,  cautions,  and  alerts 
Wearable  EEC  Development  and  Testing 
Warfighter-Machine  Interface 
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NO.  OF 

COPIES  ORGANIZATION 


1  DEFENSE  TECHNICAL 
(PDF  INFORMATION  CTR 
only)  DTIC  OCA 

8725  JOHN  J  KINGMAN  RD 
STE  0944 

FORT  BELVOIR  VA  22060-6218 

1  DIRECTOR 

US  ARMY  RESEARCH  LAB 
IMAL  HRA 

2800  POWDER  MILL  RD 
ADELPHI  MD  20783-1 197 

1  DIRECTOR 

US  ARMY  RESEARCH  LAB 
RDRL  CIO  LL 
2800  POWDER  MILL  RD 
ADELPHI  MD  20783-1 197 

1  DIRECTOR 

US  ARMY  RESEARCH  LAB 
RDRL  CIO  LT 
2800  POWDER  MILL  RD 
ADELPHI  MD  20783-1 197 
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NO.  OF 

COPIES  ORGANIZATION 


NO.  OF 

COPIES  ORGANIZATION 


1  ARMY  RSCH  LABORATORY  -  HRED 
RDRL  HRM  C  A  DAVISON 

320  MANSCEN  LOOP  STE  1 15 
FORT  LEONARD  WOOD  MO  65473 

2  ARMY  RSCH  LABORATORY  -  HRED 
RDRL  HRM  DI 

T  DAVIS 

J  HANSBERGER 

BLDG  5400  RM  C242 

REDSTONE  ARSENAL  AL  35898-7290 

1  ARMY  RSCH  LABORATORY  -  HRED 
RDRL  HRS  EA  DR  V  J  RICE 
BLDG  4011  RM217 
1750  GREELEY  RD 

FORT  SAM  HOUSTON  TX  78234-5002 

1  ARMY  RSCH  LABORATORY  -  HRED 
RDRL  HRM  DG  K  GUNN 
BLDG  333 

PICATINNY  ARSENAL  NJ  07806-5000 

1  ARMY  RSCH  LABORATORY  -  HRED 
ARMC  FIELD  ELEMENT 
RDRL  HRM  CH  C  BURNS 
THIRD  AVE  BLDG  1467B  RM  336 
FORT  KNOX  KY  40121 

1  ARMY  RSCH  LABORATORY  -  HRED 
AWC  FIELD  ELEMENT 
RDRL  HRM  DJ  D  DURBIN 
BLDG  4506  (DCD)  RM  107 
FORT  RUCKER  AL  36362-5000 

1  ARMY  RSCH  LABORATORY  -  HRED 
RDRL  HRM  CK  J  REINHART 
10125  KINGMAN  RD  BLDG  317 
FORT  BELVOIR  VA  22060-5828 

1  ARMY  RSCH  LABORATORY  -  HRED 
RDRL  HRM  AY  M  BARNES 
2520  HEALY  AVE 
STE  1172  BLDG  51005 
FORT  HUACHUCA  AZ  85613-7069 

1  ARMY  RSCH  LABORATORY  -  HRED 
RDRL  HR  MP  D  UNGV ARSKY 
POPE  HALL  BLDG  470 
BCBL  806  HARRISON  DR 
FORT  LEAVENWORTH  KS  66027-2302 


1  ARMY  RSCH  LABORATORY  -  HRED 
RDRL  HRM  DQ  MR  FLETCHER 
NATICK  SOLDIER  CTR 
AMSRD  NSC  WS  E  BLDG  3  RM  343 
NATICK  MA  01760-5020 

1  ARMY  RSCH  LABORATORY  -  HRED 
RDRL  HRM  AT  J  CHEN 
12350  RESEARCH  PKWY 
ORLANDO  EL  32826-3276 

1  ARMY  RSCH  LABORATORY  -  HRED 
RDRL  HRM  AT  C  KORTENHAUS 
12350  RESEARCH  PKWY 
ORLANDO  EL  32826 

1  ARMY  RSCH  LABORATORY  -  HRED 
RDRL  HRM  AS  C  MANASCO 
SIGNAL  TOWERS 
BLDG  29808A  RM  303A 
FORT  GORDON  GA  30905-5233 

1  ARMY  RSCH  LABORATORY  -  HRED 
RDRL  HRM  CU 
6501  E  11  MILE  RD  MS  284 
BLDG  200A  2ND  EL  RM  2104 
WARREN  MI  48397-5000 

1  ARMY  RSCH  LABORATORY  -  HRED 
FIRES  CTR  OF  EXCELLENCE 
FIELD  ELEMENT 
RDRL  HRM  AF  C  HERNANDEZ 
3040  NW  AUSTIN  RD  RM  221 
FORT  SILL  OK  73503-9043 

1  ARMY  RSCH  LABORATORY  -  HRED 
RDRL  HRM  AV  S  MIDDLEBROOKS 
91012  STATION  AVE 
FORT  HOOD  TX  76544-5073 

1  ARMY  RSCH  LABORATORY  -  HRED 
HUMAN  RSRCH  AND  ENGRNG 
DIRCTRT  MCOE  FIELD  ELEMENT 
RDRL  HRM  DW  E  REDDEN 
6450  WAY  ST 
BLDG  2839  RM310 
FORT  BENNING  GA  31905-5400 
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1  ARMY  G1 
(CD  DAPE  MR  B  KNAPP 
only)  300  ARMY  PENTAGON  RM  2C489 
WASHINGTON  DC  20310-0300 


ABERDEEN  PROVING  GROUND 

4  DIR  USARL 
RDRL  HR 
L  ALLENDER 
T  LETOWSKI 
RDRL  HRM 

P  SAVAGE-KNEPSHIELD 
RDRL  HRM  A 
B  AMREIN 
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